Nov 03 07:50:53.376061 ip-10-0-143-47 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 07:50:53.376072 ip-10-0-143-47 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 07:50:53.376079 ip-10-0-143-47 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 07:50:53.376294 ip-10-0-143-47 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 07:51:03.615979 ip-10-0-143-47 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 07:51:03.615992 ip-10-0-143-47 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bbe370d066774e1baf872ed20d1fe61f -- Nov 03 07:53:09.015945 ip-10-0-143-47 systemd[1]: Starting Kubernetes Kubelet... Nov 03 07:53:09.355387 ip-10-0-143-47 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 07:53:09.355387 ip-10-0-143-47 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 07:53:09.355387 ip-10-0-143-47 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 07:53:09.355387 ip-10-0-143-47 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 07:53:09.360427 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360335 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.360427 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360428 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.360427 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360432 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360436 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360440 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360442 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360445 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360448 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360451 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360454 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360456 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360460 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360463 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360466 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360469 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360472 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360474 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360477 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360480 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360482 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360485 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360487 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.360525 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360490 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360492 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360495 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360502 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360504 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360507 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360510 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360513 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360515 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360518 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360520 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360523 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360525 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360528 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360533 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360538 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360542 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360545 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360547 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.361003 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360550 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360556 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360559 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360561 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360564 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360567 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360570 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360572 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360575 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360577 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360580 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360583 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360585 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360587 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360590 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360593 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360596 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360600 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360603 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360606 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.361437 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360609 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360611 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360614 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360616 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360619 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360621 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360624 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360626 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360630 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360634 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360637 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360656 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360659 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360662 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360664 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360667 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360670 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360673 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360676 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.361920 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360679 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360682 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360684 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360687 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360689 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360692 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360694 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360697 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360701 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360704 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360706 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360709 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360712 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360714 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360717 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360720 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360723 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360725 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360728 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.362358 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360730 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360733 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360735 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360738 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360740 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360743 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.360746 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361548 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361555 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361559 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361563 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361566 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361569 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361572 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361574 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361577 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361579 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361582 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361585 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361588 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.362839 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361590 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361594 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361597 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361599 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361602 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361604 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361607 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361610 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361613 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361615 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361618 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361620 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361623 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361625 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361628 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361630 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361633 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361635 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361638 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361657 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.363328 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361659 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361662 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361665 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361668 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361671 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361673 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361676 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361678 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361681 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361683 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361686 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361689 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361692 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361695 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361698 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361701 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361703 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361706 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361708 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361710 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.363823 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361713 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361717 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361721 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361724 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361726 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361729 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361731 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361734 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361736 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361739 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361741 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361744 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361746 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361749 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361751 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361754 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361757 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361760 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361763 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.364303 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361765 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361768 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361770 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361773 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361779 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361781 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361784 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361787 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361789 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361792 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361794 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361797 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361800 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361803 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361805 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361808 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361811 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361815 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361819 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.364808 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361822 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361825 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361828 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361830 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361833 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361836 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361839 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361841 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361845 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361847 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361850 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361854 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361856 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361859 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.361861 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362356 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362361 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362367 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362370 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362373 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.365294 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362376 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362378 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362381 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362385 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362390 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362392 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362395 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362398 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362400 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362403 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362406 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362408 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362411 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362413 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362417 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362419 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362422 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362425 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362427 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.365783 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362430 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362432 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362435 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362437 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362440 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362443 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362445 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362448 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362450 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362453 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362457 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362460 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362462 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362465 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362467 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362470 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362472 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362475 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362478 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362480 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.366232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362483 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362485 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362488 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362490 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362492 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362495 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362498 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362500 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362503 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362506 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362508 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362511 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362513 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362516 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362518 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362521 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362523 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362527 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362529 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362532 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.366758 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362534 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362537 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362541 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362544 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362546 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362549 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362551 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362554 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362556 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362559 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362561 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362564 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362566 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362570 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362572 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362575 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362577 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362580 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362582 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.367232 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362585 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362587 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362590 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362592 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362595 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362597 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362600 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362602 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362605 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362607 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362610 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362613 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362615 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362618 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362620 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362625 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362628 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362630 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362633 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.367732 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362635 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362655 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362660 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.362663 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363332 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363343 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363349 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363354 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363359 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363362 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363367 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363371 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363375 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363386 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363391 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363394 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363397 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363400 2573 flags.go:64] FLAG: --cgroup-root="" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363403 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363406 2573 flags.go:64] FLAG: --client-ca-file="" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363409 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363412 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363417 2573 flags.go:64] FLAG: --cluster-domain="" Nov 03 07:53:09.368219 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363420 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363423 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363427 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363430 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363435 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363441 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363444 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363447 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363451 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363453 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363456 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363460 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363462 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363467 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363470 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363473 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363475 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363478 2573 flags.go:64] FLAG: --enable-server="true" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363481 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363485 2573 flags.go:64] FLAG: --event-burst="100" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363489 2573 flags.go:64] FLAG: --event-qps="50" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363492 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363495 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363497 2573 flags.go:64] FLAG: --eviction-hard="" Nov 03 07:53:09.368763 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363502 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363505 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363508 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363511 2573 flags.go:64] FLAG: --eviction-soft="" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363514 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363517 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363520 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363523 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363526 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363529 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363532 2573 flags.go:64] FLAG: --feature-gates="" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363536 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363540 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363545 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363549 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363552 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363555 2573 flags.go:64] FLAG: --help="false" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363558 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363561 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363564 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363567 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363571 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363574 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 07:53:09.369343 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363577 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363580 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363583 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363585 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363588 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363591 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363594 2573 flags.go:64] FLAG: --kube-reserved="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363597 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363600 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363603 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363605 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363608 2573 flags.go:64] FLAG: --lock-file="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363611 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363614 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363617 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363623 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363625 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363629 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363631 2573 flags.go:64] FLAG: --logging-format="text" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363634 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363638 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363655 2573 flags.go:64] FLAG: --manifest-url="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363660 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363666 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363669 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 07:53:09.369966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363672 2573 flags.go:64] FLAG: --max-pods="110" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363676 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363678 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363681 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363684 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363688 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363690 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363693 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363700 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363703 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363706 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363709 2573 flags.go:64] FLAG: --pod-cidr="" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363712 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363715 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363718 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363721 2573 flags.go:64] FLAG: --port="10250" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363724 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363727 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00c126761964cd032" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363730 2573 flags.go:64] FLAG: --qos-reserved="" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363733 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363736 2573 flags.go:64] FLAG: --register-node="true" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363738 2573 flags.go:64] FLAG: --register-with-taints="" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363742 2573 flags.go:64] FLAG: --registry-burst="10" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363745 2573 flags.go:64] FLAG: --registry-qps="5" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363748 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363750 2573 flags.go:64] FLAG: --reserved-memory="" Nov 03 07:53:09.370585 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363754 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363757 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363760 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363765 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363768 2573 flags.go:64] FLAG: --runonce="false" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363771 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363775 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363778 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363780 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363783 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363786 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363789 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363792 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363795 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363798 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363801 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363804 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363807 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363809 2573 flags.go:64] FLAG: --system-cgroups="" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363812 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363817 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363820 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363823 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363828 2573 flags.go:64] FLAG: --tls-min-version="" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363831 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 07:53:09.371207 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363834 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363836 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363839 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363842 2573 flags.go:64] FLAG: --v="2" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363847 2573 flags.go:64] FLAG: --version="false" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363851 2573 flags.go:64] FLAG: --vmodule="" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363855 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.363858 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364015 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364019 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364025 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364029 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364032 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364034 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364037 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364039 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364042 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364045 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364047 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364050 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364052 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.371852 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364055 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364057 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364060 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364063 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364065 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364068 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364071 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364073 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364076 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364078 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364081 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364083 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364086 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364088 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364091 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364093 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364096 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364099 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364101 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364104 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.372351 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364106 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364110 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364113 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364116 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364119 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364122 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364124 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364127 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364131 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364134 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364137 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364140 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364142 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364146 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364149 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364152 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364155 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364157 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364160 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.372834 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364162 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364165 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364168 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364170 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364173 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364175 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364177 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364180 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364182 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364184 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364187 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364189 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364192 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364194 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364198 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364201 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364204 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364207 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364210 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364213 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.373279 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364215 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364218 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364220 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364223 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364226 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364228 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364231 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364233 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364236 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364238 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364241 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364243 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364245 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364248 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364250 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364253 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364255 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364258 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364260 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.373795 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364263 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364265 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364268 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364270 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364273 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364275 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364278 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364282 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364284 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364287 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364289 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364296 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364299 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364302 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364304 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.364331 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 07:53:09.374937 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364380 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364384 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364387 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364390 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364392 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364395 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364398 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364400 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364403 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364405 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364408 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364411 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364413 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364416 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364418 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364421 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364423 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364426 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364428 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364431 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.375324 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364434 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364438 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364440 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364443 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364447 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364449 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364452 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364454 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364457 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364460 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364463 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364466 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364469 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364471 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364474 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364476 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364479 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364481 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364483 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.375874 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364486 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364488 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364491 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364493 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364496 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364498 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364501 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364503 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364506 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364508 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364511 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364513 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364516 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364518 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364522 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364524 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364527 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364531 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364533 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364536 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.376326 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364539 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364541 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364545 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364547 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364550 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364553 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364555 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364557 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364560 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364563 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364565 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364568 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364570 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364573 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364575 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364577 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364580 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364582 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364585 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.376831 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364588 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364591 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364593 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364596 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364598 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364600 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364603 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364607 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364609 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364612 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364616 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364618 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364623 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364626 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364629 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364632 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364635 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364654 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364658 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.377280 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364661 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364664 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364667 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364671 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364674 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364678 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364680 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364683 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.364686 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.372025 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.372043 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.372054 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.372058 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372145 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372152 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372155 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372158 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372161 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372164 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372167 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.377775 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372169 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372172 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372174 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372177 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372181 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372185 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372188 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372191 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372194 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372197 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372199 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372202 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372204 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372207 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372209 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372212 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372214 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372217 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372220 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.378231 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372223 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372225 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372228 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372242 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372244 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372247 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372249 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372252 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372255 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372257 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372260 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372262 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372264 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372267 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372270 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372272 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372274 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372277 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.378678 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372279 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372282 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372285 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372288 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372290 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372293 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372295 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372298 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372300 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372303 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372305 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372308 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372311 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372313 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372316 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372318 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372321 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372330 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372332 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372335 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.379305 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372337 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372340 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372342 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372345 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372347 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372350 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372352 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372355 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372358 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372360 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372363 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372366 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372368 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372371 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372373 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372376 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372379 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372382 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372384 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372387 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.380086 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372389 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372392 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372396 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372399 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372403 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372405 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372408 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372411 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372413 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372416 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372423 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372426 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372429 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372431 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372434 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372436 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372439 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372441 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372444 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.380843 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372446 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372449 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.372476 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372697 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372703 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372706 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372709 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372711 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372714 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372718 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372720 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372723 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372726 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372728 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372737 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372739 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.381291 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372742 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372744 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372747 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372750 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372752 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372755 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372758 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372760 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372769 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372772 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372774 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372777 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372779 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372782 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372784 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372787 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372789 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372791 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372794 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372796 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.381722 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372799 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372801 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372804 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372806 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372809 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372811 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372814 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372816 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372819 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372821 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372824 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372827 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372829 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372832 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372834 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372837 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372839 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372843 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372845 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.382182 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372848 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372850 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372858 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372861 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372864 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372866 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372869 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372872 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372874 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372877 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372879 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372882 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372884 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372886 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372889 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372891 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372894 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372896 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372899 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.382615 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372901 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372905 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372910 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372914 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372917 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372919 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372922 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372925 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372928 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372931 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372933 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372936 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372939 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372942 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372944 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372946 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372954 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372957 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372960 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.383068 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372962 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372965 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372967 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372970 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372972 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372975 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372977 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372980 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372982 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372985 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372987 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372990 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372992 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372994 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.372997 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.383512 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373000 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.373024 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373066 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373070 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373073 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373075 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373078 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373080 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373083 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373088 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373091 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373093 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373095 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373098 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373100 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373110 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:53:09.383871 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373112 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373115 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373117 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373120 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373122 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373125 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373128 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373133 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373137 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373140 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373142 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373145 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373148 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373151 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373153 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373156 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373158 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373161 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373163 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:53:09.384230 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373165 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373168 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373170 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373173 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373175 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373178 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373182 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373184 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373187 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373189 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373192 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373194 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373197 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373205 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373208 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373210 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373213 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373215 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373217 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:53:09.384750 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373220 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373222 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373225 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373227 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373230 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373232 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373236 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373239 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373242 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373244 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373246 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373250 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373252 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373255 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373257 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373259 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373262 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373264 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373267 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373271 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:53:09.385197 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373274 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373276 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373279 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373282 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373285 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373288 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373290 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373298 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373301 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373303 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373306 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373308 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373311 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373313 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373316 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373318 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373321 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373323 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373327 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373329 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:53:09.385809 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373333 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373336 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373338 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373341 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373343 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373346 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373348 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373350 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373353 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373355 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373358 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373361 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373364 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:09.373366 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.373424 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.375570 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.376378 2573 server.go:1014] "Starting client certificate rotation" Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.376504 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 07:53:09.386396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.376543 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 07:53:09.386852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.380480 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 07:53:09.386852 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.382082 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 07:53:09.389733 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.389626 2573 log.go:25] "Validated CRI v1 runtime API" Nov 03 07:53:09.392743 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.392726 2573 log.go:25] "Validated CRI v1 image API" Nov 03 07:53:09.393571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.393546 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 07:53:09.397935 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.397914 2573 fs.go:135] Filesystem UUIDs: map[176564bc-b259-4221-ba95-1b260c3bff8d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8cfddd34-5b6f-4198-b910-96cd2373e9f8:/dev/nvme0n1p4] Nov 03 07:53:09.398009 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.397935 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 07:53:09.403894 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.403782 2573 manager.go:217] Machine: {Timestamp:2026-11-03 07:53:09.402068457 +0000 UTC m=+0.286377834 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102149 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e1af23aec19137c69f8eb9b33df7 SystemUUID:ec23e1af-23ae-c191-37c6-9f8eb9b33df7 BootID:bbe370d0-6677-4e1b-af87-2ed20d1fe61f Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:20:2b:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:20:2b:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:2e:1a:79:62:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 07:53:09.404531 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.404521 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 07:53:09.404625 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.404614 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 07:53:09.406890 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.406864 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 07:53:09.407045 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.406893 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-47.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 07:53:09.407089 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.407048 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 07:53:09.407089 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.407056 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 07:53:09.407089 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.407084 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 07:53:09.407780 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.407770 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 07:53:09.408961 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.408951 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 07:53:09.413191 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413177 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 07:53:09.413382 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413360 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 07:53:09.413924 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413912 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 03 07:53:09.413962 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413932 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 07:53:09.413962 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413949 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 07:53:09.413962 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413958 2573 kubelet.go:405] "Adding apiserver pod source" Nov 03 07:53:09.414042 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.413966 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 07:53:09.414925 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.414908 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 07:53:09.418449 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.418319 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 07:53:09.418730 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.418714 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 07:53:09.423348 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.423335 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 07:53:09.424297 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424285 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 07:53:09.424332 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424315 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 07:53:09.424483 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424474 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 07:53:09.424516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424490 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 07:53:09.424516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424496 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 07:53:09.424516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424501 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 07:53:09.424516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424507 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 07:53:09.424516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424513 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 07:53:09.424985 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424976 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 07:53:09.425028 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424986 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 07:53:09.425028 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.424994 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 07:53:09.425028 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.425000 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 07:53:09.425028 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.425009 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 07:53:09.425028 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.425017 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 07:53:09.425568 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.425560 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 07:53:09.425601 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.425568 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 07:53:09.428663 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.428626 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-47.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 07:53:09.429609 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.429592 2573 server.go:1263] "Started kubelet" Nov 03 07:53:09.429753 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.429695 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 07:53:09.429837 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.429743 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 07:53:09.429879 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.429860 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 07:53:09.430418 ip-10-0-143-47 systemd[1]: Started Kubernetes Kubelet. Nov 03 07:53:09.431741 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.431717 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 03 07:53:09.431939 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.431922 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 07:53:09.432887 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.432859 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-47.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 07:53:09.432960 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.432858 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 07:53:09.435534 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.435503 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 07:53:09.435898 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.435882 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 07:53:09.435976 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.435897 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 07:53:09.436155 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.436133 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:09.436670 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.436632 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 07:53:09.436783 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.436632 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 07:53:09.436836 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.436789 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 07:53:09.436956 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.436940 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 03 07:53:09.436956 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.436952 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 03 07:53:09.445031 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445013 2573 factory.go:55] Registering systemd factory Nov 03 07:53:09.445094 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445034 2573 factory.go:223] Registration of the systemd container factory successfully Nov 03 07:53:09.445678 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445663 2573 factory.go:153] Registering CRI-O factory Nov 03 07:53:09.445678 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445680 2573 factory.go:223] Registration of the crio container factory successfully Nov 03 07:53:09.445808 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445727 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 07:53:09.445808 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445758 2573 factory.go:103] Registering Raw factory Nov 03 07:53:09.445808 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445774 2573 manager.go:1199] Started watching for new ooms in manager Nov 03 07:53:09.445974 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.445826 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pqfdn" Nov 03 07:53:09.446358 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.446104 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pqfdn" Nov 03 07:53:09.446543 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.446517 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-47.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 07:53:09.447069 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.447049 2573 manager.go:319] Starting recovery of all containers Nov 03 07:53:09.447343 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.446420 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-47.us-east-2.compute.internal.18e479b98c419a75 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-47.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-47.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 07:53:09.429320309 +0000 UTC m=+0.313629684,LastTimestamp:2026-11-03 07:53:09.429320309 +0000 UTC m=+0.313629684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-47.us-east-2.compute.internal,}" Nov 03 07:53:09.452410 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.452379 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 07:53:09.460516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.460501 2573 manager.go:324] Recovery completed Nov 03 07:53:09.464232 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.464220 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.466310 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466295 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.466358 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466324 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.466358 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466338 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.466967 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466953 2573 cpu_manager.go:226] "Starting" policy="none" Nov 03 07:53:09.466967 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466966 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 07:53:09.467053 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466988 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 07:53:09.467053 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.466956 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.467995 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.467980 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.468040 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.468006 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.468040 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.468016 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.469088 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.469077 2573 policy_none.go:50] "Start" Nov 03 07:53:09.469127 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.469092 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 07:53:09.469127 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.469101 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 07:53:09.471089 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.471077 2573 policy_none.go:44] "Start" Nov 03 07:53:09.508209 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.508178 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 07:53:09.510913 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.510896 2573 manager.go:341] "Starting Device Plugin manager" Nov 03 07:53:09.511006 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.510927 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 07:53:09.511006 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.510936 2573 server.go:87] "Starting device plugin registration server" Nov 03 07:53:09.511158 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.511147 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 07:53:09.511192 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.511160 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 07:53:09.511298 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.511272 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 07:53:09.511482 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.511467 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 07:53:09.511482 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.511483 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 07:53:09.512005 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.511976 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 07:53:09.512085 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.512026 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:09.571139 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.570977 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 07:53:09.573193 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.572436 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 07:53:09.573193 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.572456 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 07:53:09.573193 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.572486 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 07:53:09.573193 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.572519 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 07:53:09.575083 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.575065 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:53:09.611704 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.611607 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.613017 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.612999 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.613077 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.613033 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.613077 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.613043 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.613077 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.613076 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.619845 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.619830 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.619920 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.619852 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-47.us-east-2.compute.internal\": node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:09.628476 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.628459 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:09.673529 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.673489 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal"] Nov 03 07:53:09.673727 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.673570 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.674600 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.674568 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.674720 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.674611 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.674720 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.674623 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.677817 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.677793 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.677976 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.677958 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.678039 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.677999 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.678684 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.678665 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.678791 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.678696 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.678791 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.678706 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.678791 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.678672 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.678791 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.678771 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.678791 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.678785 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.680830 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.680818 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.680879 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.680842 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:53:09.682138 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.682122 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:53:09.682202 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.682148 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:53:09.682202 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.682158 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:53:09.704301 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.704281 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.706840 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.706820 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.729285 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.729259 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:09.738617 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.738592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.738730 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.738619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57afb19d8731b471ac7e459ac8d782fe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"57afb19d8731b471ac7e459ac8d782fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.738730 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.738656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.830090 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.830064 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:09.839494 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.839471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.839561 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.839492 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.839561 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.839511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.839668 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.839554 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.839668 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.839583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57afb19d8731b471ac7e459ac8d782fe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"57afb19d8731b471ac7e459ac8d782fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.839668 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:09.839616 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57afb19d8731b471ac7e459ac8d782fe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"57afb19d8731b471ac7e459ac8d782fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:09.930947 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:09.930867 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:10.008318 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.008290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:10.010445 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.010430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:10.031654 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.031625 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:10.132207 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.132173 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:10.232764 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.232706 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-47.us-east-2.compute.internal\" not found" Nov 03 07:53:10.304824 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.304795 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:53:10.336696 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.336673 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:10.346749 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.346731 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 07:53:10.348080 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.348068 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:53:10.355858 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.355846 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 07:53:10.375944 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.375926 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 07:53:10.376046 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.376025 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 07:53:10.376092 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.376028 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 07:53:10.414258 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.414232 2573 apiserver.go:52] "Watching apiserver" Nov 03 07:53:10.432127 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.432100 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 07:53:10.432452 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.432430 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-brfqb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww","openshift-ovn-kubernetes/ovnkube-node-ww6bv","kube-system/konnectivity-agent-llmvz","kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal","openshift-multus/multus-lnzwz","openshift-dns/node-resolver-nbxj7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-vfclf","openshift-network-operator/iptables-alerter-w9w62","kube-system/global-pull-secret-syncer-hqxvr","openshift-cluster-node-tuning-operator/tuned-5gvgx","openshift-image-registry/node-ca-skwds","openshift-multus/network-metrics-daemon-d7m5s"] Nov 03 07:53:10.435037 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.435021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.436396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.436378 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 07:53:10.437052 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.437037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.437128 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.437090 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:10.439182 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.439161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.439498 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.439472 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.439660 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.439628 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dc6md\"" Nov 03 07:53:10.439795 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.439781 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.441391 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.441374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.442633 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.442617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712391ec-a5a3-4824-9219-a1b56b5f214e-hosts-file\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.442720 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.442657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-kubelet-config\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.442774 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.442732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-dbus\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.442774 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.442764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.442854 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.442796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712391ec-a5a3-4824-9219-a1b56b5f214e-tmp-dir\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.442889 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.442851 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxz9h\" (UniqueName: \"kubernetes.io/projected/712391ec-a5a3-4824-9219-a1b56b5f214e-kube-api-access-kxz9h\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.443511 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.443492 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:10.443581 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.443562 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:10.445603 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.445589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.449136 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.449116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.451168 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.451150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.451405 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.451327 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 07:48:09 +0000 UTC" deadline="2028-05-05 21:56:56.973538565 +0000 UTC" Nov 03 07:53:10.451405 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.451403 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13190h3m46.522140066s" Nov 03 07:53:10.453490 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.453474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.455712 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.455700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.457896 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.457884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:10.457954 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.457928 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:10.460068 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.460048 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.483025 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.482983 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 07:53:10.483105 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.483059 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q7zwm" Nov 03 07:53:10.501063 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.501038 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q7zwm" Nov 03 07:53:10.537631 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.537608 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 07:53:10.540061 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.540040 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 07:53:10.540767 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.540743 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.540767 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.540764 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 07:53:10.540964 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.540877 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.540964 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.540749 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 07:53:10.540964 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.540929 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 07:53:10.541882 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.541847 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 07:53:10.542028 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542010 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmmvq\"" Nov 03 07:53:10.542110 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542072 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 07:53:10.542190 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542161 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 07:53:10.542303 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542271 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.542390 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542164 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9glt\"" Nov 03 07:53:10.542492 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542405 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 07:53:10.542492 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542426 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2t62\"" Nov 03 07:53:10.542709 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542673 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 07:53:10.542885 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542868 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4xzk\"" Nov 03 07:53:10.542969 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.542956 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.543096 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543082 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.543366 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vs4s\" (UniqueName: \"kubernetes.io/projected/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-kube-api-access-6vs4s\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.543366 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543340 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-systemd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.543494 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-sys\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.543548 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543492 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-thqm7\"" Nov 03 07:53:10.543548 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:10.543705 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.543705 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af81fa3b-465f-4608-bfe1-cf082d873a95-host-slash\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.543705 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-run\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.543705 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-netns\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.543827 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-ovn\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.543827 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-node-log\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.543884 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-env-overrides\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.543884 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.543943 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.543907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-socket-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.544035 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.544021 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:10.544100 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.544091 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.044061848 +0000 UTC m=+1.928371221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:10.544171 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544157 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 07:53:10.544171 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544162 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwf4\"" Nov 03 07:53:10.544270 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544198 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 07:53:10.544270 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544243 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2mmmx\"" Nov 03 07:53:10.544388 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544267 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpwxk\"" Nov 03 07:53:10.544388 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544285 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.544388 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544307 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.544388 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544325 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.544388 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:10.544388 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544376 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-k8s-cni-cncf-io\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544445 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544246 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544491 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-etc-kubernetes\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544560 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-serviceca\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.544710 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-script-lib\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-dbus\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-var-lib-kubelet\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544814 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544851 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-conf-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-systemd-units\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.545082 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-dbus\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.545488 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.544930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-tuned\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.545569 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-kubelet\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.545569 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-device-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.545677 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-modprobe-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.545677 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.545769 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-config\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.545824 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-systemd\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.545877 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss78v\" (UniqueName: \"kubernetes.io/projected/63ce7d86-82e4-4e3e-8aa7-0a131032133b-kube-api-access-ss78v\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.545877 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cni-binary-copy\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.545960 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-bin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.545960 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-netd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.546039 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.545972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712391ec-a5a3-4824-9219-a1b56b5f214e-hosts-file\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.546067 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-kubelet-config\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.546067 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxst\" (UniqueName: \"kubernetes.io/projected/ce951427-6f0a-46cc-8ac9-9776395bc3b8-kube-api-access-jwxst\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.546140 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-host\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.546140 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712391ec-a5a3-4824-9219-a1b56b5f214e-tmp-dir\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.546140 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546074 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712391ec-a5a3-4824-9219-a1b56b5f214e-hosts-file\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.546140 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546120 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-kubelet-config\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:10.546140 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-system-cni-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-kubernetes\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-system-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-multus\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-multus-certs\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-slash\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cnibin\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.546370 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-sys-fs\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bkl\" (UniqueName: \"kubernetes.io/projected/f7615d94-79a8-4c1e-ad82-c3034722c74f-kube-api-access-b2bkl\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546411 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712391ec-a5a3-4824-9219-a1b56b5f214e-tmp-dir\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysconfig\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cnibin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-daemon-config\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/728dc542-604c-46b2-9313-79f41aaf379e-konnectivity-ca\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvqn\" (UniqueName: \"kubernetes.io/projected/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-kube-api-access-lsvqn\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-log-socket\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndsf\" (UniqueName: \"kubernetes.io/projected/af81fa3b-465f-4608-bfe1-cf082d873a95-kube-api-access-7ndsf\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-host\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.546747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-tmp\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-kubelet\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-bin\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxz9h\" (UniqueName: \"kubernetes.io/projected/712391ec-a5a3-4824-9219-a1b56b5f214e-kube-api-access-kxz9h\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/728dc542-604c-46b2-9313-79f41aaf379e-agent-certs\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-etc-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mqm\" (UniqueName: \"kubernetes.io/projected/6c125505-d546-4704-9e55-ecb5b867aa4d-kube-api-access-28mqm\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.546980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jqk\" (UniqueName: \"kubernetes.io/projected/f3a66f1e-1c9c-477d-b174-5763e095ba84-kube-api-access-f8jqk\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547001 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-socket-dir-parent\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-netns\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-hostroot\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-var-lib-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-etc-selinux\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-conf\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.547250 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-lib-modules\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.547875 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c125505-d546-4704-9e55-ecb5b867aa4d-ovn-node-metrics-cert\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.547875 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-registration-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.547875 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.547280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af81fa3b-465f-4608-bfe1-cf082d873a95-iptables-alerter-script\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.553483 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.553463 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 07:53:10.557004 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.556983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxz9h\" (UniqueName: \"kubernetes.io/projected/712391ec-a5a3-4824-9219-a1b56b5f214e-kube-api-access-kxz9h\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.647543 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af81fa3b-465f-4608-bfe1-cf082d873a95-host-slash\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-run\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-netns\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.647631 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af81fa3b-465f-4608-bfe1-cf082d873a95-host-slash\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.647727 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.147707651 +0000 UTC m=+2.032017017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647706 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-ovn\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-netns\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-ovn\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.647770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647773 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-run\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-node-log\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-env-overrides\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-socket-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647852 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-node-log\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-k8s-cni-cncf-io\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-etc-kubernetes\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.647970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-serviceca\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648000 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-socket-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648027 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-etc-kubernetes\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648027 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-k8s-cni-cncf-io\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-script-lib\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-var-lib-kubelet\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.648247 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-conf-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648176 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-systemd-units\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-tuned\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-kubelet\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-device-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-modprobe-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-conf-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-env-overrides\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-config\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-systemd-units\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-systemd\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648405 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss78v\" (UniqueName: \"kubernetes.io/projected/63ce7d86-82e4-4e3e-8aa7-0a131032133b-kube-api-access-ss78v\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cni-binary-copy\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-device-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-bin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649047 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648503 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-netd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648525 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-serviceca\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxst\" (UniqueName: \"kubernetes.io/projected/ce951427-6f0a-46cc-8ac9-9776395bc3b8-kube-api-access-jwxst\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648511 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-bin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-host\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-system-cni-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-netd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648632 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-kubelet\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-kubernetes\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-system-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648761 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-multus\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648801 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-multus\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-multus-certs\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-slash\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.649928 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cnibin\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-sys-fs\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bkl\" (UniqueName: \"kubernetes.io/projected/f7615d94-79a8-4c1e-ad82-c3034722c74f-kube-api-access-b2bkl\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysconfig\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cnibin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-daemon-config\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649022 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-modprobe-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/728dc542-604c-46b2-9313-79f41aaf379e-konnectivity-ca\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-system-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvqn\" (UniqueName: \"kubernetes.io/projected/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-kube-api-access-lsvqn\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cnibin\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649258 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-systemd\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649301 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cnibin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-log-socket\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.650732 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-sys-fs\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndsf\" (UniqueName: \"kubernetes.io/projected/af81fa3b-465f-4608-bfe1-cf082d873a95-kube-api-access-7ndsf\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-host\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-tmp\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-kubelet\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649449 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-system-cni-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649464 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-multus-certs\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-bin\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-host\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.648571 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-var-lib-kubelet\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649511 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-bin\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/728dc542-604c-46b2-9313-79f41aaf379e-agent-certs\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-etc-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649569 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mqm\" (UniqueName: \"kubernetes.io/projected/6c125505-d546-4704-9e55-ecb5b867aa4d-kube-api-access-28mqm\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jqk\" (UniqueName: \"kubernetes.io/projected/f3a66f1e-1c9c-477d-b174-5763e095ba84-kube-api-access-f8jqk\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:10.651559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-socket-dir-parent\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-socket-dir-parent\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649763 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-slash\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-etc-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649416 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysconfig\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649839 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-kubelet\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649961 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-netns\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-kubernetes\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-hostroot\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-var-lib-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-etc-selinux\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650198 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-var-lib-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-conf\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.649538 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-log-socket\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-lib-modules\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.652571 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-host\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-hostroot\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650351 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-lib-modules\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650366 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650378 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-etc-selinux\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c125505-d546-4704-9e55-ecb5b867aa4d-ovn-node-metrics-cert\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650405 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-netns\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-conf\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-registration-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af81fa3b-465f-4608-bfe1-cf082d873a95-iptables-alerter-script\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650483 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-registration-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vs4s\" (UniqueName: \"kubernetes.io/projected/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-kube-api-access-6vs4s\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-systemd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-sys\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650558 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-systemd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650626 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-sys\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.650948 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.653454 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.651030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-script-lib\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.651875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-tuned\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.652046 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-config\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.652199 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af81fa3b-465f-4608-bfe1-cf082d873a95-iptables-alerter-script\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.653259 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/728dc542-604c-46b2-9313-79f41aaf379e-konnectivity-ca\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.653261 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cni-binary-copy\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.653314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.654174 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.653623 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-daemon-config\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.654720 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.654696 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-tmp\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.654809 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.654719 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c125505-d546-4704-9e55-ecb5b867aa4d-ovn-node-metrics-cert\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.655329 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.655310 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/728dc542-604c-46b2-9313-79f41aaf379e-agent-certs\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.655820 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.655804 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:10.655878 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.655825 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:10.655878 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.655835 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:10.655949 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.655881 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.155867832 +0000 UTC m=+2.040177192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:10.661501 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.661470 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jqk\" (UniqueName: \"kubernetes.io/projected/f3a66f1e-1c9c-477d-b174-5763e095ba84-kube-api-access-f8jqk\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:10.661708 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.661630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bkl\" (UniqueName: \"kubernetes.io/projected/f7615d94-79a8-4c1e-ad82-c3034722c74f-kube-api-access-b2bkl\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.662294 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.662266 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndsf\" (UniqueName: \"kubernetes.io/projected/af81fa3b-465f-4608-bfe1-cf082d873a95-kube-api-access-7ndsf\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.662368 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.662314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvqn\" (UniqueName: \"kubernetes.io/projected/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-kube-api-access-lsvqn\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.662410 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.662375 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vs4s\" (UniqueName: \"kubernetes.io/projected/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-kube-api-access-6vs4s\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.662501 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.662486 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss78v\" (UniqueName: \"kubernetes.io/projected/63ce7d86-82e4-4e3e-8aa7-0a131032133b-kube-api-access-ss78v\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.662552 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.662531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxst\" (UniqueName: \"kubernetes.io/projected/ce951427-6f0a-46cc-8ac9-9776395bc3b8-kube-api-access-jwxst\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.662583 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.662562 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mqm\" (UniqueName: \"kubernetes.io/projected/6c125505-d546-4704-9e55-ecb5b867aa4d-kube-api-access-28mqm\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.726012 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.725986 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:53:10.730810 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.730785 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57afb19d8731b471ac7e459ac8d782fe.slice/crio-9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff WatchSource:0}: Error finding container 9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff: Status 404 returned error can't find the container with id 9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff Nov 03 07:53:10.735348 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.735333 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:53:10.765367 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.765340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:53:10.775063 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.775041 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712391ec_a5a3_4824_9219_a1b56b5f214e.slice/crio-24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7 WatchSource:0}: Error finding container 24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7: Status 404 returned error can't find the container with id 24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7 Nov 03 07:53:10.785397 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.785373 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:10.793805 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.793778 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c125505_d546_4704_9e55_ecb5b867aa4d.slice/crio-86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4 WatchSource:0}: Error finding container 86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4: Status 404 returned error can't find the container with id 86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4 Nov 03 07:53:10.794804 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.794788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:10.803680 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.803656 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728dc542_604c_46b2_9313_79f41aaf379e.slice/crio-bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809 WatchSource:0}: Error finding container bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809: Status 404 returned error can't find the container with id bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809 Nov 03 07:53:10.809859 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.809840 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lnzwz" Nov 03 07:53:10.817785 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.817762 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce951427_6f0a_46cc_8ac9_9776395bc3b8.slice/crio-a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3 WatchSource:0}: Error finding container a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3: Status 404 returned error can't find the container with id a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3 Nov 03 07:53:10.824026 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.824004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:53:10.828450 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.828431 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-skwds" Nov 03 07:53:10.846767 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.846748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:53:10.847774 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.847751 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b35f75_7b9a_4a00_a775_8f2ae7cc9a34.slice/crio-4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21 WatchSource:0}: Error finding container 4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21: Status 404 returned error can't find the container with id 4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21 Nov 03 07:53:10.848396 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.848375 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30259dd3_c5f2_46b2_a5e1_acdfe2a77416.slice/crio-a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b WatchSource:0}: Error finding container a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b: Status 404 returned error can't find the container with id a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b Nov 03 07:53:10.848954 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.848933 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf590e0382a000936d43fee8841977af7.slice/crio-588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8 WatchSource:0}: Error finding container 588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8: Status 404 returned error can't find the container with id 588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8 Nov 03 07:53:10.849967 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.849948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:53:10.853953 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.853936 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:53:10.857679 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.857637 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7615d94_79a8_4c1e_ad82_c3034722c74f.slice/crio-2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb WatchSource:0}: Error finding container 2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb: Status 404 returned error can't find the container with id 2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb Nov 03 07:53:10.858421 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.858399 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf81fa3b_465f_4608_bfe1_cf082d873a95.slice/crio-15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f WatchSource:0}: Error finding container 15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f: Status 404 returned error can't find the container with id 15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f Nov 03 07:53:10.862859 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:10.862841 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ce7d86_82e4_4e3e_8aa7_0a131032133b.slice/crio-1b0a4d1863381707db6bc5d217221d2e167370d0ed991d3f7f7e60e6ee2de40f WatchSource:0}: Error finding container 1b0a4d1863381707db6bc5d217221d2e167370d0ed991d3f7f7e60e6ee2de40f: Status 404 returned error can't find the container with id 1b0a4d1863381707db6bc5d217221d2e167370d0ed991d3f7f7e60e6ee2de40f Nov 03 07:53:10.864789 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.864774 2573 kuberuntime_manager.go:1707] "Unhandled Error" err="container tuned start failed in pod tuned-5gvgx_openshift-cluster-node-tuning-operator(63ce7d86-82e4-4e3e-8aa7-0a131032133b): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 03 07:53:10.864860 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:10.864799 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" podUID="63ce7d86-82e4-4e3e-8aa7-0a131032133b" Nov 03 07:53:10.922412 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:10.922381 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:53:11.052477 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.052398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:11.052613 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.052549 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:11.052613 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.052609 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:12.052593568 +0000 UTC m=+2.936902947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:11.153128 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.153100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:11.153294 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.153212 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:11.153294 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.153268 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:12.153253387 +0000 UTC m=+3.037562750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:11.254472 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.254435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:11.254640 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.254559 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:11.254640 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.254572 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:11.254640 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.254581 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:11.254640 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.254626 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:12.25461403 +0000 UTC m=+3.138923406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:11.280202 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.280173 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:53:11.504498 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.504338 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:48:10 +0000 UTC" deadline="2028-04-13 20:17:51.129432509 +0000 UTC" Nov 03 07:53:11.504498 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.504389 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12660h24m39.625046009s" Nov 03 07:53:11.575661 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.575592 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:11.575847 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.575745 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:11.598520 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.598447 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" event={"ID":"63ce7d86-82e4-4e3e-8aa7-0a131032133b","Type":"ContainerStarted","Data":"1b0a4d1863381707db6bc5d217221d2e167370d0ed991d3f7f7e60e6ee2de40f"} Nov 03 07:53:11.605736 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:11.604048 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" podUID="63ce7d86-82e4-4e3e-8aa7-0a131032133b" Nov 03 07:53:11.607092 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.606948 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w9w62" event={"ID":"af81fa3b-465f-4608-bfe1-cf082d873a95","Type":"ContainerStarted","Data":"15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f"} Nov 03 07:53:11.612317 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.612206 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" event={"ID":"f7615d94-79a8-4c1e-ad82-c3034722c74f","Type":"ContainerStarted","Data":"2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb"} Nov 03 07:53:11.622394 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.622358 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"f590e0382a000936d43fee8841977af7","Type":"ContainerStarted","Data":"588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8"} Nov 03 07:53:11.630040 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.629941 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerStarted","Data":"4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21"} Nov 03 07:53:11.633389 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.633311 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lnzwz" event={"ID":"ce951427-6f0a-46cc-8ac9-9776395bc3b8","Type":"ContainerStarted","Data":"a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3"} Nov 03 07:53:11.639517 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.639417 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-llmvz" event={"ID":"728dc542-604c-46b2-9313-79f41aaf379e","Type":"ContainerStarted","Data":"bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809"} Nov 03 07:53:11.643502 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.643385 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4"} Nov 03 07:53:11.649300 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.649272 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-skwds" event={"ID":"30259dd3-c5f2-46b2-a5e1-acdfe2a77416","Type":"ContainerStarted","Data":"a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b"} Nov 03 07:53:11.659990 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.659671 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nbxj7" event={"ID":"712391ec-a5a3-4824-9219-a1b56b5f214e","Type":"ContainerStarted","Data":"24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7"} Nov 03 07:53:11.663365 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:11.663327 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"57afb19d8731b471ac7e459ac8d782fe","Type":"ContainerStarted","Data":"9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff"} Nov 03 07:53:12.061516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.061466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:12.061747 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.061696 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:12.061833 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.061775 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:14.061751388 +0000 UTC m=+4.946060760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:12.162127 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.162080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:12.162315 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.162296 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:12.162403 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.162373 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:14.162352618 +0000 UTC m=+5.046661985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:12.263530 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.262870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:12.263530 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.263075 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:12.263530 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.263101 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:12.263530 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.263115 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:12.263530 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.263180 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:14.263160565 +0000 UTC m=+5.147469952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:12.505176 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.505052 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:48:10 +0000 UTC" deadline="2028-08-19 19:18:03.000123274 +0000 UTC" Nov 03 07:53:12.505176 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.505124 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15731h24m50.495003604s" Nov 03 07:53:12.573681 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.572897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:12.573681 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.573033 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:12.573681 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.573511 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:12.573681 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:12.573624 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:12.985822 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:12.985785 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:53:13.575253 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:13.575228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:13.575795 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:13.575348 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:14.078132 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.078087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:14.078370 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.078340 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:14.078446 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.078421 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:18.078400105 +0000 UTC m=+8.962709470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:14.179573 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.179537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:14.179776 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.179741 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:14.179834 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.179809 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:18.179789887 +0000 UTC m=+9.064099252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:14.280909 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.280870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:14.281383 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.281096 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:14.281383 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.281117 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:14.281383 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.281131 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:14.281383 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.281190 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:18.281171646 +0000 UTC m=+9.165481010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:14.574677 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.573695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:14.574677 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.574065 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:14.574677 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.574475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:14.574677 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:14.574594 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:14.680754 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.680683 2573 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="ee611d01fe56f57b31dd71e25812c824d5846be8470834fc25bc99377b62fe1a" exitCode=0 Nov 03 07:53:14.680754 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:14.680737 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerDied","Data":"ee611d01fe56f57b31dd71e25812c824d5846be8470834fc25bc99377b62fe1a"} Nov 03 07:53:15.575385 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.573191 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:15.575385 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:15.573320 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:15.687747 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.687706 2573 generic.go:356] "Generic (PLEG): container finished" podID="f590e0382a000936d43fee8841977af7" containerID="ec75a9d8e51ec75edb764b053f0f2f2c6b07a90d2c0a2ac62188c934775b45b7" exitCode=0 Nov 03 07:53:15.688217 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.687807 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"f590e0382a000936d43fee8841977af7","Type":"ContainerDied","Data":"ec75a9d8e51ec75edb764b053f0f2f2c6b07a90d2c0a2ac62188c934775b45b7"} Nov 03 07:53:15.697981 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.697938 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-llmvz" event={"ID":"728dc542-604c-46b2-9313-79f41aaf379e","Type":"ContainerStarted","Data":"6782439e0ad221eaaf54a3d51e721eb6245b8b7931302edeaec42ac5f062885d"} Nov 03 07:53:15.708517 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.707422 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"57afb19d8731b471ac7e459ac8d782fe","Type":"ContainerStarted","Data":"5371d854841dfdf78511ea866f68d6a57d654b870c4decfc8412c6f46b6be4a7"} Nov 03 07:53:15.735674 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.735550 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-llmvz" podStartSLOduration=2.171920299 podStartE2EDuration="6.735531171s" totalImagesPullingTime="4.563610872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:15.720157255 +0000 UTC m=+6.604466639" watchObservedRunningTime="2026-11-03 07:53:15.735531171 +0000 UTC m=+6.619840557" Nov 03 07:53:15.736177 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:15.735918 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" podStartSLOduration=5.73590548 podStartE2EDuration="5.73590548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:15.734821897 +0000 UTC m=+6.619131281" watchObservedRunningTime="2026-11-03 07:53:15.73590548 +0000 UTC m=+6.620214861" Nov 03 07:53:16.573945 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:16.573172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:16.573945 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:16.573323 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:16.573945 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:16.573736 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:16.573945 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:16.573848 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:16.714387 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:16.713681 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"f590e0382a000936d43fee8841977af7","Type":"ContainerStarted","Data":"e55369473eae6ae134090499d3d5e06198fa8c936779b9b9fe9fafa58eee2465"} Nov 03 07:53:16.718678 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:16.718437 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-skwds" event={"ID":"30259dd3-c5f2-46b2-a5e1-acdfe2a77416","Type":"ContainerStarted","Data":"76b10aafb852da2457a2fabb88d087d38e0c8efdbd19054a458edb114243bfb9"} Nov 03 07:53:16.739715 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:16.739657 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" podStartSLOduration=6.739625341 podStartE2EDuration="6.739625341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:16.73880046 +0000 UTC m=+7.623109844" watchObservedRunningTime="2026-11-03 07:53:16.739625341 +0000 UTC m=+7.623934726" Nov 03 07:53:17.217132 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:17.217049 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:17.217894 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:17.217872 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:17.236264 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:17.236200 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-skwds" podStartSLOduration=2.893598468 podStartE2EDuration="8.236181007s" totalImagesPullingTime="5.342582539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:16.755950032 +0000 UTC m=+7.640259415" watchObservedRunningTime="2026-11-03 07:53:17.236181007 +0000 UTC m=+8.120490409" Nov 03 07:53:17.573840 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:17.573772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:17.573975 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:17.573913 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:17.725170 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:17.725133 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" event={"ID":"f7615d94-79a8-4c1e-ad82-c3034722c74f","Type":"ContainerStarted","Data":"decea2349f267269e0d631d08b9c9851cf0e23549db38cce944b3da60ed30f20"} Nov 03 07:53:18.112570 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.112532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:18.112770 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.112675 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:18.112770 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.112751 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:26.112733203 +0000 UTC m=+16.997042587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:18.213695 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.213655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:18.213880 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.213863 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:18.213946 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.213933 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:26.213914908 +0000 UTC m=+17.098224272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:18.315056 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.315010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:18.315234 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.315200 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:18.315234 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.315233 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:18.315339 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.315246 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:18.315339 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.315316 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:26.315295766 +0000 UTC m=+17.199605139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:18.572849 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.572820 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:18.573010 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.572950 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:18.573359 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.573338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:18.573474 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:18.573447 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:18.598516 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.598480 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 07:53:18.730539 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.730448 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" event={"ID":"f7615d94-79a8-4c1e-ad82-c3034722c74f","Type":"ContainerStarted","Data":"53ae3ab98a4caed8407d091d22175abf195bf506b0ab5d84b9ef8e866dda1766"} Nov 03 07:53:18.733161 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.733138 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 07:53:18.733309 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.733173 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lnzwz" event={"ID":"ce951427-6f0a-46cc-8ac9-9776395bc3b8","Type":"ContainerStarted","Data":"2eba8b9f780e7c2491b37c02269c2c9bd567d3177c2ffcc83923329174bad514"} Nov 03 07:53:18.758003 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:18.757940 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lnzwz" podStartSLOduration=2.073385114 podStartE2EDuration="9.757921676s" totalImagesPullingTime="7.684536562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:18.753443002 +0000 UTC m=+9.637752409" watchObservedRunningTime="2026-11-03 07:53:18.757921676 +0000 UTC m=+9.642231061" Nov 03 07:53:19.516925 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.516829 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T07:53:18.598495417Z","UUID":"a55046a4-b357-4418-b6c4-46918b2cfe68","Handler":null,"Name":"","Endpoint":""} Nov 03 07:53:19.523821 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.523792 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 07:53:19.523821 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.523829 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 07:53:19.573659 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.573618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:19.573830 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:19.573736 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:19.738799 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.738768 2573 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="8e15b08b88e3e4e8001e1d01e60247c4593308ab04fdc3ce266eea0c8b693db3" exitCode=0 Nov 03 07:53:19.739152 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.739048 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerDied","Data":"8e15b08b88e3e4e8001e1d01e60247c4593308ab04fdc3ce266eea0c8b693db3"} Nov 03 07:53:19.741231 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.741208 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nbxj7" event={"ID":"712391ec-a5a3-4824-9219-a1b56b5f214e","Type":"ContainerStarted","Data":"33c2ad0f620cc579e2dfb1bfb7205640f9b831b58a2ecbc3c520d300df8a72cb"} Nov 03 07:53:19.774333 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:19.774285 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nbxj7" podStartSLOduration=2.737624502 podStartE2EDuration="10.774268807s" totalImagesPullingTime="8.036644305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:19.774239359 +0000 UTC m=+10.658548744" watchObservedRunningTime="2026-11-03 07:53:19.774268807 +0000 UTC m=+10.658578178" Nov 03 07:53:20.573736 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.573696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:20.573878 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.573706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:20.573878 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:20.573838 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:20.574009 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:20.573879 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:20.746215 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.746180 2573 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="7a60eb848962a4b4b7140ff206a71a56de1443115fe249f379d302efb2be516e" exitCode=0 Nov 03 07:53:20.746624 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.746275 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerDied","Data":"7a60eb848962a4b4b7140ff206a71a56de1443115fe249f379d302efb2be516e"} Nov 03 07:53:20.748051 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.748028 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w9w62" event={"ID":"af81fa3b-465f-4608-bfe1-cf082d873a95","Type":"ContainerStarted","Data":"30a9dc311c7a4177c49a31ebe5198ff4601218af50562f7208ebad1ab8f871cd"} Nov 03 07:53:20.750533 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.750512 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" event={"ID":"f7615d94-79a8-4c1e-ad82-c3034722c74f","Type":"ContainerStarted","Data":"2fdb7c9c53b606a280d6305bd60e490799c77a1cf080d6e312d2efeb396e9d18"} Nov 03 07:53:20.781552 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.781507 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" podStartSLOduration=3.299375992 podStartE2EDuration="11.781496425s" totalImagesPullingTime="8.482120433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:20.781305044 +0000 UTC m=+11.665614427" watchObservedRunningTime="2026-11-03 07:53:20.781496425 +0000 UTC m=+11.665805807" Nov 03 07:53:20.796117 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:20.796073 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w9w62" podStartSLOduration=3.71013268 podStartE2EDuration="11.796057952s" totalImagesPullingTime="8.085925272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:20.795505734 +0000 UTC m=+11.679815137" watchObservedRunningTime="2026-11-03 07:53:20.796057952 +0000 UTC m=+11.680367338" Nov 03 07:53:21.573106 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:21.572876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:21.573335 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:21.573207 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:22.572839 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:22.572795 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:22.573320 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:22.572795 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:22.573320 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:22.572919 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:22.573320 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:22.573020 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:23.404213 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:23.404187 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbxj7_712391ec-a5a3-4824-9219-a1b56b5f214e/dns-node-resolver/0.log" Nov 03 07:53:23.573488 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:23.573451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:23.574025 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:23.573571 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:23.985894 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:23.985864 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skwds_30259dd3-c5f2-46b2-a5e1-acdfe2a77416/node-ca/0.log" Nov 03 07:53:24.573053 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:24.573022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:24.573226 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:24.573127 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:24.573226 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:24.573192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:24.573329 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:24.573303 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:25.573621 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:25.573589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:25.574198 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:25.573862 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:25.760878 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:25.760851 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"948d4bb5221ece75623b72d3fd187e0b4b64c454e67f8d73f8082be1ec326103"} Nov 03 07:53:26.174155 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.173970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:26.174284 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.174114 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:26.174284 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.174239 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:42.174224262 +0000 UTC m=+33.058533623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:26.275095 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.275074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:26.275204 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.275191 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:26.275250 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.275242 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:42.275226453 +0000 UTC m=+33.159535832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:26.376329 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.376241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:26.376483 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.376389 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:26.376483 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.376408 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:26.376483 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.376421 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:26.376483 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.376475 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:42.376458409 +0000 UTC m=+33.260767787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:26.573611 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.573572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:26.573788 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.573583 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:26.573788 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.573750 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:26.574276 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:26.573847 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:26.765614 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.765518 2573 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="6af5eab8b0fe3c90617543d77d3aecb145874dcd83913315c95efba5a60f1b03" exitCode=0 Nov 03 07:53:26.765614 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.765595 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerDied","Data":"6af5eab8b0fe3c90617543d77d3aecb145874dcd83913315c95efba5a60f1b03"} Nov 03 07:53:26.768435 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768358 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:53:26.768669 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768627 2573 generic.go:356] "Generic (PLEG): container finished" podID="6c125505-d546-4704-9e55-ecb5b867aa4d" containerID="6ac15cff00f326810ca67ca0999755dde71bdf870a3659a005e6682d0b7b7495" exitCode=1 Nov 03 07:53:26.768669 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768662 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"944c2dfc5be76d03670bfde11c896e2a832c5ddad8217b5a09c304c3931b249a"} Nov 03 07:53:26.768839 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768781 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"0587bbe8ca8d8c749ee1d547afc2603ce7e730c1c95982e389350f73e9e98591"} Nov 03 07:53:26.768894 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768841 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"d10b0f2be76927c01b12117efeedb63275a8fbc43a4f4c928b144d5c557a6e7e"} Nov 03 07:53:26.768949 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768890 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"99e0f8e1994f118168595ded7f8a472f107dceab7361493efbbfe4d85699fdb0"} Nov 03 07:53:26.768949 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.768939 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerDied","Data":"6ac15cff00f326810ca67ca0999755dde71bdf870a3659a005e6682d0b7b7495"} Nov 03 07:53:26.797444 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.797423 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:26.797547 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.797530 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 07:53:26.797970 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:26.797953 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:53:27.573338 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:27.573142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:27.573523 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:27.573427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:28.572883 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:28.572843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:28.573304 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:28.572952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:28.573304 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:28.573012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:28.573304 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:28.573139 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:28.778427 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:28.778405 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:53:28.778830 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:28.778808 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"058277b395d19fcb20c758e59529f7e8c8e0a1ae405c8b42e4cf45707cee1488"} Nov 03 07:53:28.781396 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:28.781371 2573 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="3c761dbf2bbf2c394b018b0d16edcd807fc994b19727145f0e00e7696a1f78a2" exitCode=0 Nov 03 07:53:28.781477 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:28.781405 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerDied","Data":"3c761dbf2bbf2c394b018b0d16edcd807fc994b19727145f0e00e7696a1f78a2"} Nov 03 07:53:29.574112 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:29.574083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:29.574495 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:29.574159 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:29.787836 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:29.787801 2573 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="1695981a9ea7bcafce00c302f656b518c40bc81563531bc2e47ea41f7f11a442" exitCode=0 Nov 03 07:53:29.787990 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:29.787865 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerDied","Data":"1695981a9ea7bcafce00c302f656b518c40bc81563531bc2e47ea41f7f11a442"} Nov 03 07:53:30.573596 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.573576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:30.573715 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:30.573699 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:30.573768 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.573750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:30.573847 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:30.573833 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:30.797056 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.797021 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfclf" event={"ID":"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34","Type":"ContainerStarted","Data":"68d030885f99bf4a31cf9453dfad82b325d92180b90a8ed75f81a67d8ddc3afa"} Nov 03 07:53:30.800604 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.800581 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:53:30.800987 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.800949 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"8fc5dcb7b7c5e9f871f185e4b80dd3a02672c938b4027a6e4ab3feaa74e2c149"} Nov 03 07:53:30.801236 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.801203 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:30.801335 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.801272 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:30.801335 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.801322 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:30.801442 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.801347 2573 scope.go:122] "RemoveContainer" containerID="6ac15cff00f326810ca67ca0999755dde71bdf870a3659a005e6682d0b7b7495" Nov 03 07:53:30.820044 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.820007 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:30.822060 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.822012 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vfclf" podStartSLOduration=6.953202585 podStartE2EDuration="21.821996078s" totalImagesPullingTime="14.868793493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:30.820437817 +0000 UTC m=+21.704747200" watchObservedRunningTime="2026-11-03 07:53:30.821996078 +0000 UTC m=+21.706305463" Nov 03 07:53:30.823136 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:30.823105 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:53:31.573324 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:31.573289 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:31.573612 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:31.573411 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:31.807892 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:31.807864 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:53:31.808400 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:31.808345 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" event={"ID":"6c125505-d546-4704-9e55-ecb5b867aa4d","Type":"ContainerStarted","Data":"030ad4f044649ce3a5c1390afd6ebf66460a8df482bc8315fdd7271f54a0c02e"} Nov 03 07:53:31.837879 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:31.837783 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" podStartSLOduration=7.973646679 podStartE2EDuration="22.83776961s" totalImagesPullingTime="14.864122931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:31.835986659 +0000 UTC m=+22.720296041" watchObservedRunningTime="2026-11-03 07:53:31.83776961 +0000 UTC m=+22.722078992" Nov 03 07:53:32.573501 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:32.573453 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:32.573501 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:32.573481 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:32.573728 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:32.573570 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:32.573769 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:32.573724 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:33.573304 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:33.573264 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:33.573702 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:33.573370 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:33.813968 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:33.813787 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" event={"ID":"63ce7d86-82e4-4e3e-8aa7-0a131032133b","Type":"ContainerStarted","Data":"6bdb09a0580316376978744b00f4acc10ec52fc2f86799103970697ccfa4d688"} Nov 03 07:53:33.830665 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:33.830552 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" podStartSLOduration=2.08098181 podStartE2EDuration="24.830541532s" totalImagesPullingTime="22.749559722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-03 07:53:33.830190661 +0000 UTC m=+24.714500035" watchObservedRunningTime="2026-11-03 07:53:33.830541532 +0000 UTC m=+24.714850914" Nov 03 07:53:34.573610 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:34.573584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:34.573996 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:34.573702 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:34.573996 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:34.573750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:34.573996 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:34.573835 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:35.576030 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:35.576001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:35.576389 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:35.576095 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:36.573172 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:36.573140 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:36.573404 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:36.573273 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:36.573404 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:36.573319 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:36.573521 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:36.573399 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:37.579197 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:37.579167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:37.579580 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:37.579265 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:38.381066 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:38.380988 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d7m5s"] Nov 03 07:53:38.381284 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:38.381126 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:38.381284 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:38.381243 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:38.383974 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:38.383935 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hqxvr"] Nov 03 07:53:38.384280 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:38.384264 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:38.384513 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:38.384490 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:38.384809 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:38.384787 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-brfqb"] Nov 03 07:53:38.384891 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:38.384869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:38.385011 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:38.384990 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:39.183198 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:39.183173 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbxj7_712391ec-a5a3-4824-9219-a1b56b5f214e/dns-node-resolver/0.log" Nov 03 07:53:39.576184 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:39.576162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:39.576338 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:39.576248 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:39.982863 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:39.982791 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skwds_30259dd3-c5f2-46b2-a5e1-acdfe2a77416/node-ca/0.log" Nov 03 07:53:40.573119 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:40.573080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:40.573525 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:40.573097 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:40.573525 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:40.573195 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-brfqb" podUID="db268603-dbe4-4ca3-8f27-b4e27414f5e1" Nov 03 07:53:40.573525 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:40.573263 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7m5s" podUID="f3a66f1e-1c9c-477d-b174-5763e095ba84" Nov 03 07:53:41.575861 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:41.575689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:41.576211 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:41.575935 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxvr" podUID="abfaf52a-6e21-4290-93cf-367a0aa9d749" Nov 03 07:53:42.186018 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.185926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:42.186153 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.186089 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:42.186194 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.186159 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret podName:abfaf52a-6e21-4290-93cf-367a0aa9d749 nodeName:}" failed. No retries permitted until 2026-11-03 07:54:14.186142053 +0000 UTC m=+65.070451432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret") pod "global-pull-secret-syncer-hqxvr" (UID: "abfaf52a-6e21-4290-93cf-367a0aa9d749") : object "kube-system"/"original-pull-secret" not registered Nov 03 07:53:42.286480 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.286443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:42.286629 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.286597 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:42.286690 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.286678 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs podName:f3a66f1e-1c9c-477d-b174-5763e095ba84 nodeName:}" failed. No retries permitted until 2026-11-03 07:54:14.286661908 +0000 UTC m=+65.170971269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs") pod "network-metrics-daemon-d7m5s" (UID: "f3a66f1e-1c9c-477d-b174-5763e095ba84") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 07:53:42.387624 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.387594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:42.387784 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.387723 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 07:53:42.387784 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.387736 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 07:53:42.387784 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.387745 2573 projected.go:196] Error preparing data for projected volume kube-api-access-n9nfk for pod openshift-network-diagnostics/network-check-target-brfqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:42.387876 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:53:42.387789 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk podName:db268603-dbe4-4ca3-8f27-b4e27414f5e1 nodeName:}" failed. No retries permitted until 2026-11-03 07:54:14.387778064 +0000 UTC m=+65.272087424 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9nfk" (UniqueName: "kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk") pod "network-check-target-brfqb" (UID: "db268603-dbe4-4ca3-8f27-b4e27414f5e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 07:53:42.443841 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.443779 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeReady" Nov 03 07:53:42.443939 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.443906 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 07:53:42.494496 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.494469 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zqtl5"] Nov 03 07:53:42.515000 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.514965 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j4bwh"] Nov 03 07:53:42.515126 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.515098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.518904 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.518881 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 07:53:42.519267 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.519248 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjbhz\"" Nov 03 07:53:42.519351 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.519304 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 07:53:42.531125 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.531107 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zqtl5"] Nov 03 07:53:42.531198 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.531173 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j4bwh"] Nov 03 07:53:42.531240 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.531207 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pp56g"] Nov 03 07:53:42.531308 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.531293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.549167 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.549150 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pp56g"] Nov 03 07:53:42.549251 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.549241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.572901 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.572883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:53:42.573071 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.573061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:53:42.588703 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.588684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5497a21d-0e9d-4191-bf7b-392785b052d0-tmp-dir\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.589086 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.588717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-cert\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.589086 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.588736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5497a21d-0e9d-4191-bf7b-392785b052d0-config-volume\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.589086 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.588782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xgg\" (UniqueName: \"kubernetes.io/projected/5497a21d-0e9d-4191-bf7b-392785b052d0-kube-api-access-44xgg\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.589086 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.588870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8p78\" (UniqueName: \"kubernetes.io/projected/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-kube-api-access-s8p78\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.589086 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.588906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5497a21d-0e9d-4191-bf7b-392785b052d0-metrics-tls\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.619456 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619437 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 07:53:42.619548 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619498 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 07:53:42.619602 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619587 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 07:53:42.619676 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619602 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 07:53:42.619820 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619805 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 07:53:42.619911 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619813 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 07:53:42.619911 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619857 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ghck6\"" Nov 03 07:53:42.619911 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.619879 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5spf6\"" Nov 03 07:53:42.620832 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.620819 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\"" Nov 03 07:53:42.621230 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.621217 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 07:53:42.621287 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.621253 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 07:53:42.621287 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.621271 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 07:53:42.621381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.621294 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\"" Nov 03 07:53:42.621381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.621303 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 07:53:42.689876 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.689846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8p78\" (UniqueName: \"kubernetes.io/projected/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-kube-api-access-s8p78\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.689876 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.689881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-crio-socket\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.690055 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.690141 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5497a21d-0e9d-4191-bf7b-392785b052d0-metrics-tls\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.690180 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5497a21d-0e9d-4191-bf7b-392785b052d0-tmp-dir\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.690224 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-cert\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.690257 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5497a21d-0e9d-4191-bf7b-392785b052d0-config-volume\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.690365 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xgg\" (UniqueName: \"kubernetes.io/projected/5497a21d-0e9d-4191-bf7b-392785b052d0-kube-api-access-44xgg\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.690422 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkjn\" (UniqueName: \"kubernetes.io/projected/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-api-access-lvkjn\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.690473 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-data-volume\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.690473 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690451 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.690595 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690576 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5497a21d-0e9d-4191-bf7b-392785b052d0-tmp-dir\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.690908 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.690891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5497a21d-0e9d-4191-bf7b-392785b052d0-config-volume\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.694227 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.694187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5497a21d-0e9d-4191-bf7b-392785b052d0-metrics-tls\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.694279 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.694246 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-cert\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.699206 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.699187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xgg\" (UniqueName: \"kubernetes.io/projected/5497a21d-0e9d-4191-bf7b-392785b052d0-kube-api-access-44xgg\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.699384 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.699368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8p78\" (UniqueName: \"kubernetes.io/projected/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-kube-api-access-s8p78\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.791421 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkjn\" (UniqueName: \"kubernetes.io/projected/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-api-access-lvkjn\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.791421 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-data-volume\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.791690 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.791690 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-crio-socket\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.791690 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791476 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.791832 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791759 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-crio-socket\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.791832 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.791808 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-data-volume\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.792165 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.792142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.793982 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.793964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.799521 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.799494 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkjn\" (UniqueName: \"kubernetes.io/projected/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-api-access-lvkjn\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:42.850175 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.850139 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:42.868315 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.868288 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:53:42.922625 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:42.922594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:53:43.228450 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.228428 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j4bwh"] Nov 03 07:53:43.231475 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.231455 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zqtl5"] Nov 03 07:53:43.232199 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:43.232172 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f0df62_afcf_4d4f_ae18_ddcdbdc99bba.slice/crio-6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c WatchSource:0}: Error finding container 6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c: Status 404 returned error can't find the container with id 6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c Nov 03 07:53:43.233501 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:43.233473 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5497a21d_0e9d_4191_bf7b_392785b052d0.slice/crio-cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e WatchSource:0}: Error finding container cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e: Status 404 returned error can't find the container with id cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e Nov 03 07:53:43.234742 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.234722 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pp56g"] Nov 03 07:53:43.236721 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:43.236702 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba44899_0b4f_4ae4_86c0_a6e4657c9c73.slice/crio-0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad WatchSource:0}: Error finding container 0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad: Status 404 returned error can't find the container with id 0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad Nov 03 07:53:43.572912 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.572731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:53:43.581363 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.581342 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\"" Nov 03 07:53:43.581452 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.581372 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 07:53:43.837038 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.836955 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pp56g" event={"ID":"fba44899-0b4f-4ae4-86c0-a6e4657c9c73","Type":"ContainerStarted","Data":"b92c3b55200e825640c404c3b2243553e4a4e67aed6116e16e65db6b32d9fd68"} Nov 03 07:53:43.837038 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.837022 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pp56g" event={"ID":"fba44899-0b4f-4ae4-86c0-a6e4657c9c73","Type":"ContainerStarted","Data":"0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad"} Nov 03 07:53:43.838469 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.838436 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j4bwh" event={"ID":"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba","Type":"ContainerStarted","Data":"6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c"} Nov 03 07:53:43.839696 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:43.839664 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqtl5" event={"ID":"5497a21d-0e9d-4191-bf7b-392785b052d0","Type":"ContainerStarted","Data":"cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e"} Nov 03 07:53:44.843959 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:44.843886 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pp56g" event={"ID":"fba44899-0b4f-4ae4-86c0-a6e4657c9c73","Type":"ContainerStarted","Data":"8d9f32c5700bf5249c68574fe3acc352b144af8bc97a3ee91d9cc70cde23f517"} Nov 03 07:53:45.568906 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.568875 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj"] Nov 03 07:53:45.588786 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.588752 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.595129 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.595103 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vtg6x"] Nov 03 07:53:45.608270 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.608245 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj"] Nov 03 07:53:45.608389 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.608375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.614214 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.614188 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 07:53:45.614450 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.614431 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 07:53:45.614625 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.614611 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8pp7x\"" Nov 03 07:53:45.659032 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.659000 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 07:53:45.659032 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.659019 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8t4tj\"" Nov 03 07:53:45.659243 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.659050 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 07:53:45.659243 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.659135 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 07:53:45.659243 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.659190 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 07:53:45.659243 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.659192 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 07:53:45.660546 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.660524 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 07:53:45.660691 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.660600 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 07:53:45.711180 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce8c5d5b-8c05-4057-a77d-4a102186c208-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.711180 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.711381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-root\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-wtmp\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-sys\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711381 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711372 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvz6\" (UniqueName: \"kubernetes.io/projected/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-api-access-plvz6\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-metrics-client-ca\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcfs\" (UniqueName: \"kubernetes.io/projected/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-kube-api-access-bjcfs\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-textfile\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.711581 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.711565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-tls\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.811921 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.811888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-sys\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.811921 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.811925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812113 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.811959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.812113 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.811981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvz6\" (UniqueName: \"kubernetes.io/projected/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-api-access-plvz6\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.812113 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-metrics-client-ca\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812113 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcfs\" (UniqueName: \"kubernetes.io/projected/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-kube-api-access-bjcfs\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812113 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.812113 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-textfile\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-tls\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce8c5d5b-8c05-4057-a77d-4a102186c208-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-root\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-wtmp\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812340 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.812971 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.812946 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.813059 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.813030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-sys\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.814235 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.814212 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.814327 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.814254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-root\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.814327 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.814315 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce8c5d5b-8c05-4057-a77d-4a102186c208-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.814474 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.814454 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-textfile\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.814590 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.814572 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-wtmp\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.814918 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.814891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.815096 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.815074 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-metrics-client-ca\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.817313 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.817287 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.817590 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.817570 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-tls\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.820062 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.819971 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.822673 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.822433 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvz6\" (UniqueName: \"kubernetes.io/projected/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-api-access-plvz6\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:45.823721 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.822987 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.824579 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.824539 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcfs\" (UniqueName: \"kubernetes.io/projected/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-kube-api-access-bjcfs\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:45.957662 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:45.957619 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:53:46.015319 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.013546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:53:46.257963 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.257709 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj"] Nov 03 07:53:46.261026 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:46.260984 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8c5d5b_8c05_4057_a77d_4a102186c208.slice/crio-4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768 WatchSource:0}: Error finding container 4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768: Status 404 returned error can't find the container with id 4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768 Nov 03 07:53:46.849826 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.849785 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtg6x" event={"ID":"84e304c5-3d08-4c75-8589-1ee0e67c4ed6","Type":"ContainerStarted","Data":"10e03d381d24c47cb6a799361ea2490821d33ae5556d7fc9df24836211d5096f"} Nov 03 07:53:46.854118 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.854083 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" event={"ID":"ce8c5d5b-8c05-4057-a77d-4a102186c208","Type":"ContainerStarted","Data":"4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768"} Nov 03 07:53:46.855713 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.855694 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqtl5" event={"ID":"5497a21d-0e9d-4191-bf7b-392785b052d0","Type":"ContainerStarted","Data":"bfd8f91bf582e20e66539841d2986240c55d5339f24d22ea504a2f6c8c30cd27"} Nov 03 07:53:46.855757 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.855729 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqtl5" event={"ID":"5497a21d-0e9d-4191-bf7b-392785b052d0","Type":"ContainerStarted","Data":"f67b818e604e61db885c4c904c46f6cc37139b0c0347c12f4c99d4b7b4e2e387"} Nov 03 07:53:46.855921 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.855906 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:46.873559 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:46.873516 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zqtl5" podStartSLOduration=2.280137855 podStartE2EDuration="4.873504422s" totalImagesPullingTime="2.593366567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:46.87174694 +0000 UTC m=+37.756056342" watchObservedRunningTime="2026-11-03 07:53:46.873504422 +0000 UTC m=+37.757813825" Nov 03 07:53:47.646145 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.646115 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f889895df-lctqt"] Nov 03 07:53:47.660850 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.660820 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f889895df-lctqt"] Nov 03 07:53:47.660974 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.660931 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.665366 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665227 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 07:53:47.665366 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665276 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 07:53:47.665366 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665304 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 07:53:47.665366 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665316 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 07:53:47.665584 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665389 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-17ekhfl0cqidc\"" Nov 03 07:53:47.665584 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665395 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6cpzp\"" Nov 03 07:53:47.665584 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.665402 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 07:53:47.730622 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.730826 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-tmp\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.730826 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.730946 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-metrics-client-ca\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.730946 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhgc\" (UniqueName: \"kubernetes.io/projected/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-kube-api-access-4nhgc\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.730946 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.730946 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.730908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-grpc-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.731099 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.731076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.731158 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.731141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.831966 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.831940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832087 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.831984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832087 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832087 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-tmp\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832087 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832337 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-metrics-client-ca\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832337 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhgc\" (UniqueName: \"kubernetes.io/projected/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-kube-api-access-4nhgc\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832488 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832467 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-tmp\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832628 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.832628 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.832635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-grpc-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.833089 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.833063 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-metrics-client-ca\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.834749 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.834722 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.834855 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.834728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.835066 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.835037 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.835152 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.835042 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.835412 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.835392 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.835776 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.835758 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-grpc-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.839761 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.839741 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhgc\" (UniqueName: \"kubernetes.io/projected/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-kube-api-access-4nhgc\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:47.862728 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.862697 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pp56g" event={"ID":"fba44899-0b4f-4ae4-86c0-a6e4657c9c73","Type":"ContainerStarted","Data":"ca20661868014e9c55aa6fe94c051d744f75559e2ac2dc2853b718d3311f8bc8"} Nov 03 07:53:47.864603 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.864578 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j4bwh" event={"ID":"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba","Type":"ContainerStarted","Data":"e29598e99f36a35afebacc21d47564308b0ca230af03db3d13464841834df4bf"} Nov 03 07:53:47.880715 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.880669 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pp56g" podStartSLOduration=2.411724696 podStartE2EDuration="5.880631158s" totalImagesPullingTime="3.468906462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:47.879412734 +0000 UTC m=+38.763722137" watchObservedRunningTime="2026-11-03 07:53:47.880631158 +0000 UTC m=+38.764940540" Nov 03 07:53:47.896720 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:47.896187 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j4bwh" podStartSLOduration=1.885205445 podStartE2EDuration="5.89616936s" totalImagesPullingTime="4.010963915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:47.895138754 +0000 UTC m=+38.779448137" watchObservedRunningTime="2026-11-03 07:53:47.89616936 +0000 UTC m=+38.780478743" Nov 03 07:53:48.031733 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.031689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:48.324059 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.324038 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f889895df-lctqt"] Nov 03 07:53:48.325816 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:48.325794 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff1dba0_28ed_4e7f_ae34_9d30f7b881a1.slice/crio-1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5 WatchSource:0}: Error finding container 1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5: Status 404 returned error can't find the container with id 1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5 Nov 03 07:53:48.872148 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.872113 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" event={"ID":"ce8c5d5b-8c05-4057-a77d-4a102186c208","Type":"ContainerStarted","Data":"dce2f51f5ba807313c4f6c5c5cde86dfc9cc7586e4169dc063015e20b01ac927"} Nov 03 07:53:48.872479 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.872172 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" event={"ID":"ce8c5d5b-8c05-4057-a77d-4a102186c208","Type":"ContainerStarted","Data":"933ef752b3384a82dd6bda33679e86393b8bd5b8d2bcabfb76c36a8687f4eb6f"} Nov 03 07:53:48.873374 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.873344 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5"} Nov 03 07:53:48.874684 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.874657 2573 generic.go:356] "Generic (PLEG): container finished" podID="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" containerID="9599b122983c8b1892b30766e4bd5768ecbaab714365ef328b6dd989328a9f5e" exitCode=0 Nov 03 07:53:48.874785 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:48.874741 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtg6x" event={"ID":"84e304c5-3d08-4c75-8589-1ee0e67c4ed6","Type":"ContainerDied","Data":"9599b122983c8b1892b30766e4bd5768ecbaab714365ef328b6dd989328a9f5e"} Nov 03 07:53:49.879380 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:49.879338 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" event={"ID":"ce8c5d5b-8c05-4057-a77d-4a102186c208","Type":"ContainerStarted","Data":"5fdf301b0ab88f3d89fc417a9a855753786e2a676e1605e71fd9afb142c26456"} Nov 03 07:53:49.881453 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:49.881428 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtg6x" event={"ID":"84e304c5-3d08-4c75-8589-1ee0e67c4ed6","Type":"ContainerStarted","Data":"4a821af86ad52289a38b7a67a601db668112a4d51dc1b5ba2fdad5c8b2dff397"} Nov 03 07:53:49.881594 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:49.881470 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtg6x" event={"ID":"84e304c5-3d08-4c75-8589-1ee0e67c4ed6","Type":"ContainerStarted","Data":"066e0c65c838af42e132829ed26f80ba8c10d493bac8b2897a7561c36f7cc342"} Nov 03 07:53:49.900668 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:49.900597 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" podStartSLOduration=2.521990454 podStartE2EDuration="4.900583377s" totalImagesPullingTime="2.378592923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:49.899538166 +0000 UTC m=+40.783847545" watchObservedRunningTime="2026-11-03 07:53:49.900583377 +0000 UTC m=+40.784892760" Nov 03 07:53:49.919401 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:49.919353 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vtg6x" podStartSLOduration=3.098913279 podStartE2EDuration="4.919341559s" totalImagesPullingTime="1.730752694s" totalInitContainerRuntime="89.675586ms" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:49.91891857 +0000 UTC m=+40.803227976" watchObservedRunningTime="2026-11-03 07:53:49.919341559 +0000 UTC m=+40.803650941" Nov 03 07:53:50.360794 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.360755 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh"] Nov 03 07:53:50.362950 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.362925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.367011 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.366985 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 07:53:50.367127 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.367108 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m644k\"" Nov 03 07:53:50.372204 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.372176 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh"] Nov 03 07:53:50.455299 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.455262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmjv\" (UniqueName: \"kubernetes.io/projected/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-kube-api-access-5gmjv\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.455466 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.455332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.556283 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.556244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.556464 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.556322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmjv\" (UniqueName: \"kubernetes.io/projected/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-kube-api-access-5gmjv\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.558954 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.558925 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.566015 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.565987 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmjv\" (UniqueName: \"kubernetes.io/projected/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-kube-api-access-5gmjv\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.674499 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.674410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:50.946029 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:50.946006 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh"] Nov 03 07:53:50.947972 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:50.947943 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8ebc2e_2b44_4837_b0ed_b9a4a897d52b.slice/crio-b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d WatchSource:0}: Error finding container b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d: Status 404 returned error can't find the container with id b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d Nov 03 07:53:51.851563 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.851528 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:53:51.866965 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.866939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.869962 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.869936 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:53:51.871342 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.871318 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 07:53:51.871464 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.871413 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 07:53:51.871512 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.871470 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 07:53:51.872736 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.872673 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 07:53:51.872736 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.872705 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 07:53:51.872736 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.872720 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 07:53:51.872953 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.872912 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 07:53:51.873002 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.872978 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8f9uviit3bo6o\"" Nov 03 07:53:51.873034 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.873006 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 07:53:51.873094 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.873051 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 07:53:51.874166 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.873301 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 07:53:51.874166 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.873358 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 07:53:51.874166 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.873480 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 07:53:51.874166 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.873677 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 07:53:51.874166 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.874124 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bxcxf\"" Nov 03 07:53:51.881285 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.881266 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 07:53:51.883292 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.883273 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 07:53:51.890426 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.890392 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" event={"ID":"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b","Type":"ContainerStarted","Data":"b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d"} Nov 03 07:53:51.897874 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.897845 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"2687428b8dc61514e96fa1d0edfcf2bb3153a09a748bb5a345d59c272316ca17"} Nov 03 07:53:51.897975 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.897924 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"69655a681ec8b1a700dc902956e5cc288deae53ad298d8bc03ce0d739af95b7d"} Nov 03 07:53:51.897975 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.897966 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"8bb152ee361de64dc79190beccf4e0dc62830fb171082b366198da01fa244ac0"} Nov 03 07:53:51.967601 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.967991 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.967992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968193 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968325 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:51.968540 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:51.968343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069592 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069592 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.069857 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.069960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070183 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070634 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.070634 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.070408 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.071172 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.071139 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.071567 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.071545 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.071726 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.071704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.071982 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.071953 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.072489 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.072460 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.072746 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.072695 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.074148 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.074085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.075257 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.075216 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.075687 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.075664 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.076848 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.076629 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.076848 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.076775 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.077344 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.077288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.077537 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.077518 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.078492 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.078448 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.078492 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.078455 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.079027 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.079006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.080005 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.079983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.080099 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.080013 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.081285 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.080504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.198304 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.198229 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:53:52.499855 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.499828 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:53:52.503409 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:53:52.503377 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902ad59c_068f_42e8_aa8d_d1658a31e3bf.slice/crio-4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc WatchSource:0}: Error finding container 4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc: Status 404 returned error can't find the container with id 4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc Nov 03 07:53:52.908246 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.908206 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"97b365f52eec4c84d401fd580a3467083f98bf10cba3ab27b7e22a390d073c37"} Nov 03 07:53:52.908413 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.908313 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"5acb479f4fe6feac8b3b41583ce72c7c196c2c4d35932c3155391760d0971a4f"} Nov 03 07:53:52.908413 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.908359 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:53:52.908413 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.908399 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" event={"ID":"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1","Type":"ContainerStarted","Data":"6ea32469345eb2aa1876b1796db0fd295b0a62916b4564ae0ee036ac0bf321eb"} Nov 03 07:53:52.909652 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.909622 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" event={"ID":"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b","Type":"ContainerStarted","Data":"f48c5b9b82f512885cae4755a0f3075fb68bb8b321143c2fbbbac04519d13e70"} Nov 03 07:53:52.909816 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.909800 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:52.910855 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.910830 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc"} Nov 03 07:53:52.915020 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.915000 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:53:52.936475 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.936432 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" podStartSLOduration=2.109808489 podStartE2EDuration="5.936422084s" totalImagesPullingTime="3.826613595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:52.93597768 +0000 UTC m=+43.820287088" watchObservedRunningTime="2026-11-03 07:53:52.936422084 +0000 UTC m=+43.820731466" Nov 03 07:53:52.954131 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:52.954089 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" podStartSLOduration=1.301726516 podStartE2EDuration="2.954076854s" totalImagesPullingTime="1.652350338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:52.953501788 +0000 UTC m=+43.837811186" watchObservedRunningTime="2026-11-03 07:53:52.954076854 +0000 UTC m=+43.838386237" Nov 03 07:53:53.915327 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:53.915291 2573 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" exitCode=0 Nov 03 07:53:53.915811 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:53.915370 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} Nov 03 07:53:56.870021 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:56.869986 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zqtl5" Nov 03 07:53:57.934541 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.934465 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} Nov 03 07:53:57.934541 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.934534 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} Nov 03 07:53:57.934945 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.934561 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} Nov 03 07:53:57.934945 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.934592 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} Nov 03 07:53:57.934945 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.934617 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} Nov 03 07:53:57.934945 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.934665 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerStarted","Data":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} Nov 03 07:53:57.966824 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:57.966752 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.454967924 podStartE2EDuration="6.966728683s" totalImagesPullingTime="4.494990386s" totalInitContainerRuntime="16.770373ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:57.96491484 +0000 UTC m=+48.849224249" watchObservedRunningTime="2026-11-03 07:53:57.966728683 +0000 UTC m=+48.851038068" Nov 03 07:53:58.922044 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:53:58.922016 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:02.199593 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:02.199547 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:02.822618 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:02.822593 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:12.948813 ip-10-0-143-47 kubenswrapper[2573]: E1103 07:54:12.948777 2573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:54:14.260214 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.260167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:14.263473 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.263456 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 07:54:14.272770 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.272747 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:14.360890 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.360854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:14.364025 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.363999 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 07:54:14.373942 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.373910 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:14.461619 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.461574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:14.464464 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.464442 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 07:54:14.475131 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.475112 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 07:54:14.480924 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.480902 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\"" Nov 03 07:54:14.485724 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.485704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:14.488430 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.488415 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:14.502975 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.502954 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\"" Nov 03 07:54:14.511014 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.510969 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:14.761988 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.761931 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d7m5s"] Nov 03 07:54:14.763372 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.763351 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\"" Nov 03 07:54:14.764442 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:54:14.764417 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a66f1e_1c9c_477d_b174_5763e095ba84.slice/crio-2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14 WatchSource:0}: Error finding container 2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14: Status 404 returned error can't find the container with id 2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14 Nov 03 07:54:14.771052 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.771034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:14.785686 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.785669 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hqxvr"] Nov 03 07:54:14.787860 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:54:14.787830 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfaf52a_6e21_4290_93cf_367a0aa9d749.slice/crio-0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c WatchSource:0}: Error finding container 0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c: Status 404 returned error can't find the container with id 0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c Nov 03 07:54:14.997810 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.997775 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hqxvr" event={"ID":"abfaf52a-6e21-4290-93cf-367a0aa9d749","Type":"ContainerStarted","Data":"0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c"} Nov 03 07:54:14.998899 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:14.998874 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d7m5s" event={"ID":"f3a66f1e-1c9c-477d-b174-5763e095ba84","Type":"ContainerStarted","Data":"2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14"} Nov 03 07:54:15.039683 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:15.039659 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-brfqb"] Nov 03 07:54:15.041480 ip-10-0-143-47 kubenswrapper[2573]: W1103 07:54:15.041457 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb268603_dbe4_4ca3_8f27_b4e27414f5e1.slice/crio-ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078 WatchSource:0}: Error finding container ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078: Status 404 returned error can't find the container with id ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078 Nov 03 07:54:16.005870 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:16.005829 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-brfqb" event={"ID":"db268603-dbe4-4ca3-8f27-b4e27414f5e1","Type":"ContainerStarted","Data":"ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078"} Nov 03 07:54:17.012230 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:17.012198 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d7m5s" event={"ID":"f3a66f1e-1c9c-477d-b174-5763e095ba84","Type":"ContainerStarted","Data":"9dee3a6bdd858f636f6b0540cd6f1d3cf6d92886ccb7d39d7114904286f59174"} Nov 03 07:54:17.012593 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:17.012256 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d7m5s" event={"ID":"f3a66f1e-1c9c-477d-b174-5763e095ba84","Type":"ContainerStarted","Data":"e2e41520afe70b90a71c0b62f1eab4aa5abe8000167ab2d13356e197f322cd76"} Nov 03 07:54:17.034678 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:17.031672 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d7m5s" podStartSLOduration=66.511338694 podStartE2EDuration="1m8.031634737s" totalImagesPullingTime="1.520296043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:17.029255977 +0000 UTC m=+67.913565362" watchObservedRunningTime="2026-11-03 07:54:17.031634737 +0000 UTC m=+67.915944117" Nov 03 07:54:20.026007 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:20.025970 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-brfqb" event={"ID":"db268603-dbe4-4ca3-8f27-b4e27414f5e1","Type":"ContainerStarted","Data":"3e3586492099123592497b065bb2c1f49f568d1564bc044866083f461877ae33"} Nov 03 07:54:20.026464 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:20.026139 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:20.043242 ip-10-0-143-47 kubenswrapper[2573]: I1103 07:54:20.043192 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-brfqb" podStartSLOduration=66.517048073 podStartE2EDuration="1m11.043179959s" totalImagesPullingTime="4.526131886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:20.041265845 +0000 UTC m=+70.925575228" watchObservedRunningTime="2026-11-03 07:54:20.043179959 +0000 UTC m=+70.927489341" Nov 03 07:54:20.567580 ip-10-0-143-47 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 07:54:20.576156 ip-10-0-143-47 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 07:54:20.576327 ip-10-0-143-47 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 07:54:20.576505 ip-10-0-143-47 systemd[1]: kubelet.service: Consumed 4.103s CPU time. Nov 03 07:54:20.586484 ip-10-0-143-47 systemd[1]: Starting Kubernetes Kubelet... Nov 03 07:54:20.648403 ip-10-0-143-47 kubenswrapper[5951]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 07:54:20.648403 ip-10-0-143-47 kubenswrapper[5951]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 07:54:20.648403 ip-10-0-143-47 kubenswrapper[5951]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 07:54:20.648403 ip-10-0-143-47 kubenswrapper[5951]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 07:54:20.650447 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650363 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.650447 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650447 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.650447 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650451 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650455 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650458 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650461 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650464 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650467 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650469 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650478 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650481 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650483 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650486 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650489 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650491 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650494 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650497 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650499 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650502 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650504 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650507 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650510 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.650542 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650513 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650515 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650518 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650520 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650523 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650526 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650528 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650531 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650533 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650536 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650538 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650541 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650543 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650547 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650549 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650552 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650555 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650558 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650561 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.651049 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650563 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650566 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650569 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650572 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650574 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650577 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650580 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650582 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650585 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650588 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650590 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650593 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650595 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650598 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650600 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650603 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650605 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650608 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650610 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650613 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.651525 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650615 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650620 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650624 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650627 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650630 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650633 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650636 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650679 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650682 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650685 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650687 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650690 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650693 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650696 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650698 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650701 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650704 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650706 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650710 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.652034 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650714 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650718 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650721 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650723 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650726 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650729 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650731 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650734 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650736 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650739 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650742 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650744 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650747 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650749 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650752 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650754 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650757 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650761 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650763 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.652538 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650767 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650769 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650772 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650775 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650777 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650781 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.650791 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651549 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651555 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651558 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651561 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651563 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651568 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651572 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651576 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651579 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651581 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651584 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651587 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.653092 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651589 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651592 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651595 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651597 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651600 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651602 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651605 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651608 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651610 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651613 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651616 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651619 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651622 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651626 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651628 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651631 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651633 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651636 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651654 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651657 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.653564 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651664 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651678 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651681 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651684 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651686 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651689 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651691 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651694 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651696 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651699 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651702 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651704 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651707 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651709 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651712 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651714 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651717 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651719 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651722 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.654098 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651724 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651727 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651729 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651732 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651735 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651738 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651741 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651744 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651748 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651752 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651755 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651757 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651760 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651763 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651766 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651769 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651771 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651774 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651777 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.654712 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651779 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651782 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651784 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651787 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651790 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651792 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651794 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651797 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651799 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651802 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651805 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651807 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651810 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651812 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651815 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651817 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651820 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651823 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651826 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651828 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.655189 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651832 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651834 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651837 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651839 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651842 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651845 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651847 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651850 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651855 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651857 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651860 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651863 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651866 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651869 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651872 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.651874 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652349 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652354 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652356 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652359 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.655704 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652362 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652365 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652367 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652370 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652373 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652375 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652378 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652381 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652383 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652386 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652388 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652391 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652394 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652396 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652399 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652401 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652404 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652406 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652409 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.656247 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652411 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652414 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652417 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652419 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652422 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652425 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652428 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652430 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652433 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652435 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652438 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652440 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652443 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652445 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652448 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652451 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652453 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652456 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652458 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.656763 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652460 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652463 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652465 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652469 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652473 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652476 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652479 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652482 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652484 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652487 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652490 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652492 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652495 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652497 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652500 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652503 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652506 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652513 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652515 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652518 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.657415 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652521 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652524 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652526 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652529 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652532 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652534 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652537 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652540 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652543 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652546 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652549 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652551 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652554 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652557 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652559 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652562 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652564 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652567 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652570 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652572 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.658253 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652575 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652578 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652580 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652583 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652585 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652590 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652594 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652597 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652601 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652606 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652609 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652612 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652615 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652617 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652620 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652622 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652625 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652627 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652630 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.658772 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652632 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652635 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652637 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652654 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.652656 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652755 5951 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652762 5951 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652768 5951 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652773 5951 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652777 5951 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652783 5951 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652787 5951 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652791 5951 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652794 5951 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652797 5951 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652801 5951 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652804 5951 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652807 5951 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652810 5951 flags.go:64] FLAG: --cgroup-root="" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652813 5951 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652816 5951 flags.go:64] FLAG: --client-ca-file="" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652819 5951 flags.go:64] FLAG: --cloud-provider="external" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652823 5951 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 07:54:20.659239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652828 5951 flags.go:64] FLAG: --cluster-domain="" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652831 5951 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652834 5951 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652837 5951 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652840 5951 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652844 5951 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652847 5951 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652850 5951 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652853 5951 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652856 5951 flags.go:64] FLAG: --contention-profiling="false" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652859 5951 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652862 5951 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652865 5951 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652868 5951 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652872 5951 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652875 5951 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652878 5951 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652882 5951 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652885 5951 flags.go:64] FLAG: --enable-server="true" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652889 5951 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652893 5951 flags.go:64] FLAG: --event-burst="100" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652896 5951 flags.go:64] FLAG: --event-qps="50" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652899 5951 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652902 5951 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 07:54:20.659809 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652905 5951 flags.go:64] FLAG: --eviction-hard="" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652909 5951 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652912 5951 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652915 5951 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652918 5951 flags.go:64] FLAG: --eviction-soft="" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652921 5951 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652924 5951 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652929 5951 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652932 5951 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652935 5951 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652938 5951 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652941 5951 flags.go:64] FLAG: --feature-gates="" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652945 5951 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652948 5951 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652951 5951 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652954 5951 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652957 5951 flags.go:64] FLAG: --healthz-port="10248" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652960 5951 flags.go:64] FLAG: --help="false" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652963 5951 flags.go:64] FLAG: --hostname-override="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652966 5951 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652969 5951 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652971 5951 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652975 5951 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 07:54:20.660407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652978 5951 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652981 5951 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652984 5951 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652986 5951 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652991 5951 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652994 5951 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.652997 5951 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653000 5951 flags.go:64] FLAG: --kube-reserved="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653003 5951 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653005 5951 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653008 5951 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653011 5951 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653014 5951 flags.go:64] FLAG: --lock-file="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653016 5951 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653019 5951 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653022 5951 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653029 5951 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653031 5951 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653035 5951 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653038 5951 flags.go:64] FLAG: --logging-format="text" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653040 5951 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653044 5951 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653047 5951 flags.go:64] FLAG: --manifest-url="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653050 5951 flags.go:64] FLAG: --manifest-url-header="" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653054 5951 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 07:54:20.660980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653057 5951 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653061 5951 flags.go:64] FLAG: --max-pods="110" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653063 5951 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653066 5951 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653069 5951 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653073 5951 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653076 5951 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653079 5951 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653082 5951 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653088 5951 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653091 5951 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653095 5951 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653099 5951 flags.go:64] FLAG: --pod-cidr="" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653102 5951 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653105 5951 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653108 5951 flags.go:64] FLAG: --pods-per-core="0" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653110 5951 flags.go:64] FLAG: --port="10250" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653114 5951 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653116 5951 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00c126761964cd032" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653119 5951 flags.go:64] FLAG: --qos-reserved="" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653122 5951 flags.go:64] FLAG: --read-only-port="10255" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653125 5951 flags.go:64] FLAG: --register-node="true" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653128 5951 flags.go:64] FLAG: --register-with-taints="" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653133 5951 flags.go:64] FLAG: --registry-burst="10" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653136 5951 flags.go:64] FLAG: --registry-qps="5" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653139 5951 flags.go:64] FLAG: --reserved-cpus="" Nov 03 07:54:20.661627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653143 5951 flags.go:64] FLAG: --reserved-memory="" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653147 5951 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653150 5951 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653153 5951 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653155 5951 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653158 5951 flags.go:64] FLAG: --runonce="false" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653161 5951 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653164 5951 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653167 5951 flags.go:64] FLAG: --seccomp-default="false" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653170 5951 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653172 5951 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653175 5951 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653178 5951 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653181 5951 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653184 5951 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653187 5951 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653189 5951 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653194 5951 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653197 5951 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653200 5951 flags.go:64] FLAG: --system-cgroups="" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653202 5951 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653207 5951 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653210 5951 flags.go:64] FLAG: --tls-cert-file="" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653213 5951 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653217 5951 flags.go:64] FLAG: --tls-min-version="" Nov 03 07:54:20.662299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653220 5951 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653223 5951 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653226 5951 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653228 5951 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653232 5951 flags.go:64] FLAG: --v="2" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653236 5951 flags.go:64] FLAG: --version="false" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653240 5951 flags.go:64] FLAG: --vmodule="" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653244 5951 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653248 5951 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653382 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653386 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653388 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653391 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653394 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653396 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653399 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653401 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653404 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653407 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653409 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653411 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.663089 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653414 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653417 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653419 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653424 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653426 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653429 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653431 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653434 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653437 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653439 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653442 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653444 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653446 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653449 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653452 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653456 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653459 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653461 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653465 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.663614 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653469 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653473 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653476 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653478 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653481 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653484 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653486 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653489 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653492 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653494 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653498 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653502 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653505 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653507 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653510 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653512 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653516 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653519 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653521 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653524 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.664097 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653527 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653529 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653532 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653534 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653537 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653539 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653542 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653544 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653548 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653551 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653554 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653556 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653559 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653561 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653564 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653567 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653570 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653572 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653575 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.664587 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653578 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653580 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653583 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653585 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653588 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653590 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653593 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653595 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653598 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653602 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653604 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653607 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653610 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653612 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653615 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653617 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653619 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653622 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653624 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653627 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.665129 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653629 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653632 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653635 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653637 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653658 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653662 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653666 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653670 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653677 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653680 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653684 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653686 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653688 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653691 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653694 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653696 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.665758 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.653724 5951 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653762 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653767 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653770 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653773 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653777 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653780 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653782 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653785 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653787 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653790 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653792 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653795 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653797 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653800 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653802 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653805 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653807 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653810 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.666284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653813 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653815 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653818 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653820 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653823 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653825 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653828 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653831 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653833 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653836 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653838 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653840 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653843 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653846 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653848 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653851 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653853 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653855 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653859 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.667186 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653862 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653864 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653867 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653870 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653872 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653875 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653877 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653880 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653882 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653885 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653887 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653890 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653892 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653895 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653897 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653900 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653903 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653905 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653908 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653910 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.667972 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653913 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653916 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653918 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653921 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653923 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653926 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653929 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653931 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653934 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653936 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653939 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653943 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653946 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653949 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653952 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653954 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653957 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653959 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653962 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653964 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.668637 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653966 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653969 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653971 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653974 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653976 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653979 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653982 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653984 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653987 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653991 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653994 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653997 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.653999 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654004 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654007 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654009 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654012 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654015 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654017 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.669236 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654020 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654024 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654028 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654031 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654036 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654039 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654041 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654044 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654046 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.654049 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.659314 5951 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.659331 5951 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.659349 5951 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.659354 5951 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659425 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659430 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659433 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659436 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659439 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.669727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659441 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659444 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659447 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659450 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659452 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659455 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659458 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659460 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659463 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659465 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659468 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659474 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659477 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659480 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659483 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659486 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659488 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659491 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659494 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659497 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.670199 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659499 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659501 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659505 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659508 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659510 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659513 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659515 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659518 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659520 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659523 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659525 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659528 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659531 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659533 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659536 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659538 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659541 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659543 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659546 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659549 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.670703 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659551 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659554 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659557 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659560 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659562 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659565 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659567 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659570 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659572 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659575 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659580 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659583 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659587 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659590 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659593 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659595 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659598 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659601 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659603 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.671181 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659606 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659608 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659611 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659614 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659616 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659619 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659621 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659624 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659661 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659666 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659669 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659672 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659675 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659678 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659680 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659683 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659686 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659690 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659694 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.671634 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659697 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659700 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659702 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659705 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659709 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659711 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659714 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659716 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659719 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659722 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659724 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659727 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659729 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659732 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659735 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659737 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659740 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659742 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659745 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659748 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.672113 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659750 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659752 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659755 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.659782 5951 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659919 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659924 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659927 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659930 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659932 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659935 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659938 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659941 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659944 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659946 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659949 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659952 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.672592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659955 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659958 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659960 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659963 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659965 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659968 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659971 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659973 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659976 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659978 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659981 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659983 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659986 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659988 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659991 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659993 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659996 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.659998 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660003 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660007 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.673021 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660009 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660012 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660015 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660018 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660021 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660024 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660028 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660031 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660034 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660036 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660039 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660042 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660044 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660047 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660049 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660052 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660054 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660057 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660059 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.673503 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660062 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660064 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660066 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660069 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660071 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660074 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660076 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660079 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660081 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660084 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660086 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660089 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660091 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660094 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660096 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660099 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660101 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660104 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660106 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.674004 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660108 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660111 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660114 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660117 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660119 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660122 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660124 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660127 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660129 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660132 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660134 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660137 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660139 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660141 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660144 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660146 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660149 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660151 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660154 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660156 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.674464 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660159 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660161 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660163 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660166 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660169 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660171 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660174 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660176 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660179 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660181 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660184 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660186 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660189 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660191 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660194 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.675052 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660197 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.660221 5951 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660264 5951 feature_gate.go:386] unrecognized feature gate: Example Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660269 5951 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660272 5951 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660275 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660277 5951 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660280 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660282 5951 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660285 5951 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660287 5951 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660290 5951 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660292 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660295 5951 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660297 5951 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660300 5951 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 07:54:20.675414 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660302 5951 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660305 5951 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660309 5951 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660313 5951 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660315 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660318 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660320 5951 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660323 5951 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660325 5951 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660328 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660330 5951 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660333 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660335 5951 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660338 5951 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660340 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660343 5951 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660346 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660349 5951 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660352 5951 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 07:54:20.675817 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660354 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660357 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660359 5951 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660362 5951 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660364 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660367 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660369 5951 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660371 5951 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660374 5951 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660376 5951 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660379 5951 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660381 5951 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660384 5951 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660386 5951 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660388 5951 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660392 5951 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660395 5951 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660397 5951 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660400 5951 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660403 5951 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 07:54:20.676284 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660405 5951 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660408 5951 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660411 5951 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660413 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660416 5951 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660419 5951 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660421 5951 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660424 5951 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660426 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660429 5951 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660432 5951 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660436 5951 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660441 5951 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660444 5951 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660447 5951 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660450 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660453 5951 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660455 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660458 5951 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 07:54:20.676939 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660462 5951 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660465 5951 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660468 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660471 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660473 5951 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660476 5951 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660478 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660481 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660484 5951 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660486 5951 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660489 5951 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660491 5951 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660494 5951 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660496 5951 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660499 5951 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660501 5951 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660504 5951 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660506 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660508 5951 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660511 5951 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 07:54:20.677720 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660513 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660516 5951 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660518 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660520 5951 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660524 5951 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660527 5951 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660530 5951 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660533 5951 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660535 5951 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660538 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660540 5951 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660543 5951 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660545 5951 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:20.660547 5951 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.660605 5951 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.661563 5951 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.661626 5951 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.661844 5951 server.go:1014] "Starting client certificate rotation" Nov 03 07:54:20.678295 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.661982 5951 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.662030 5951 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 07:48:09 +0000 UTC" deadline="2028-07-08 17:46:50.29233012 +0000 UTC" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.662067 5951 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14721h52m29.630264741s" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.662334 5951 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.664222 5951 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.665661 5951 log.go:25] "Validated CRI v1 runtime API" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.667948 5951 log.go:25] "Validated CRI v1 image API" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.668675 5951 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 07:54:20.678900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.670703 5951 fs.go:135] Filesystem UUIDs: map[176564bc-b259-4221-ba95-1b260c3bff8d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8cfddd34-5b6f-4198-b910-96cd2373e9f8:/dev/nvme0n1p4] Nov 03 07:54:20.679124 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.670725 5951 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c/userdata/shm major:0 minor:396 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10e03d381d24c47cb6a799361ea2490821d33ae5556d7fc9df24836211d5096f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10e03d381d24c47cb6a799361ea2490821d33ae5556d7fc9df24836211d5096f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14/userdata/shm major:0 minor:394 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34/volumes/kubernetes.io~projected/kube-api-access-6vs4s:{mountpoint:/var/lib/kubelet/pods/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34/volumes/kubernetes.io~projected/kube-api-access-6vs4s major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/volumes/kubernetes.io~projected/kube-api-access-5gmjv:{mountpoint:/var/lib/kubelet/pods/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/volumes/kubernetes.io~projected/kube-api-access-5gmjv major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30259dd3-c5f2-46b2-a5e1-acdfe2a77416/volumes/kubernetes.io~projected/kube-api-access-lsvqn:{mountpoint:/var/lib/kubelet/pods/30259dd3-c5f2-46b2-a5e1-acdfe2a77416/volumes/kubernetes.io~projected/kube-api-access-lsvqn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5497a21d-0e9d-4191-bf7b-392785b052d0/volumes/kubernetes.io~projected/kube-api-access-44xgg:{mountpoint:/var/lib/kubelet/pods/5497a21d-0e9d-4191-bf7b-392785b052d0/volumes/kubernetes.io~projected/kube-api-access-44xgg major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5497a21d-0e9d-4191-bf7b-392785b052d0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/5497a21d-0e9d-4191-bf7b-392785b052d0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~projected/kube-api-access-ss78v:{mountpoint:/var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~projected/kube-api-access-ss78v major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volumes/kubernetes.io~projected/kube-api-access-28mqm:{mountpoint:/var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volumes/kubernetes.io~projected/kube-api-access-28mqm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712391ec-a5a3-4824-9219-a1b56b5f214e/volumes/kubernetes.io~projected/kube-api-access-kxz9h:{mountpoint:/var/lib/kubelet/pods/712391ec-a5a3-4824-9219-a1b56b5f214e/volumes/kubernetes.io~projected/kube-api-access-kxz9h major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/728dc542-604c-46b2-9313-79f41aaf379e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/728dc542-604c-46b2-9313-79f41aaf379e/volumes/kubernetes.io~secret/agent-certs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~projected/kube-api-access-bjcfs:{mountpoint:/var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~projected/kube-api-access-bjcfs major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/config-reloader/2 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/prometheus/15 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~empty-dir/config-out major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~projected/kube-api-access-r5wjs:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~projected/kube-api-access-r5wjs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~projected/tls-assets major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/config major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/web-config major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abfaf52a-6e21-4290-93cf-367a0aa9d749/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/abfaf52a-6e21-4290-93cf-367a0aa9d749/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af81fa3b-465f-4608-bfe1-cf082d873a95/volumes/kubernetes.io~projected/kube-api-access-7ndsf:{mountpoint:/var/lib/kubelet/pods/af81fa3b-465f-4608-bfe1-cf082d873a95/volumes/kubernetes.io~projected/kube-api-access-7ndsf major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~projected/kube-api-access-4nhgc:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~projected/kube-api-access-4nhgc major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~projected/kube-api-access-plvz6:{mountpoint:/var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~projected/kube-api-access-plvz6 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce951427-6f0a-46cc-8ac9-9776395bc3b8/volumes/kubernetes.io~projected/kube-api-access-jwxst:{mountpoint:/var/lib/kubelet/pods/ce951427-6f0a-46cc-8ac9-9776395bc3b8/volumes/kubernetes.io~projected/kube-api-access-jwxst major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db268603-dbe4-4ca3-8f27-b4e27414f5e1/volumes/kubernetes.io~projected/kube-api-access-n9nfk:{mountpoint:/var/lib/kubelet/pods/db268603-dbe4-4ca3-8f27-b4e27414f5e1/volumes/kubernetes.io~projected/kube-api-access-n9nfk major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/volumes/kubernetes.io~projected/kube-api-access-s8p78:{mountpoint:/var/lib/kubelet/pods/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/volumes/kubernetes.io~projected/kube-api-access-s8p78 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3a66f1e-1c9c-477d-b174-5763e095ba84/volumes/kubernetes.io~projected/kube-api-access-f8jqk:{mountpoint:/var/lib/kubelet/pods/f3a66f1e-1c9c-477d-b174-5763e095ba84/volumes/kubernetes.io~projected/kube-api-access-f8jqk major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3a66f1e-1c9c-477d-b174-5763e095ba84/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f3a66f1e-1c9c-477d-b174-5763e095ba84/volumes/kubernetes.io~secret/metrics-certs major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7615d94-79a8-4c1e-ad82-c3034722c74f/volumes/kubernetes.io~projected/kube-api-access-b2bkl:{mountpoint:/var/lib/kubelet/pods/f7615d94-79a8-4c1e-ad82-c3034722c74f/volumes/kubernetes.io~projected/kube-api-access-b2bkl major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba44899-0b4f-4ae4-86c0-a6e4657c9c73/volumes/kubernetes.io~projected/kube-api-access-lvkjn:{mountpoint:/var/lib/kubelet/pods/fba44899-0b4f-4ae4-86c0-a6e4657c9c73/volumes/kubernetes.io~projected/kube-api-access-lvkjn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba44899-0b4f-4ae4-86c0-a6e4657c9c73/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/fba44899-0b4f-4ae4-86c0-a6e4657c9c73/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/29dbf020694cf1a55dcdf1eba33c2e862c1158cac5474c89cf88bc991c540bd8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a810a3178033c58737dfeeca944d6201c776c332e9314250ee1e22e1ff8adfd4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/344c290d48d803a0d4f44000190da756ac4c2fa0ac7fdb5fc305536bf2cd57be/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/324bdf8674d15f2fa8011da22db4b079d6b49b10a1d8ab0d78a3d8c1cea2ebe8/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f00be8eb748d750dea89431d58994423396846a0bdea89c01b6c884ae15d3c1e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f27f4fdf7c5cc33e209b56a62fb13c9184df0eeed86b67e8be7f85ccb7a90dbb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e83c32abb6df02b9d4d309f252137b4c12a35932d222b1283e5ea23ff8699b38/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/acc966e414efccf60496966ab6395357d80eeadb2c42dc2e3e381dc9c25046eb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/119cd9b68569919fcad26992b520eef923d2724d00fb42cacc0eb445d51874f8/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a106d550b331f696e6ff4db845bfeee9c086a5476b0729248b896333c34803f9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/0d97f099f204a728f3d87549fd3f6c12d2ccb86cd60ea63c985bcd4847d1c3d5/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8dc2cc2a51aef14117a25f1279a8902052b70455aaff93a8917d9c24572588ac/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/77cbe1a5e7b434aa11c7f60bcb5cda0fce081e53c175ba6ec3abbd5c8f87ec5d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/603d76bf69201a594bb5f7eaa4b416df82cefc411b8448eadaa9476cb642f283/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5e18f37e9f1d0d9e57239ac0390254bdc4748bf87e4553e7e41161904b0fd84a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5be66901404ee8a1216b1f9733e248a8fbf41ae10ee32c85cb8b5c2807c14558/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/60a575bf4ced906007808f102cada0692805a6f96293ceefb574372eaef0c6ee/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/5c243adf4341a2394fdced1cae17a875d1b41f069070b20d392167e44e892a0b/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/7137d85160cecea242b0fd234f15ddc7862916e8724ea7d10515ba4143c6cef1/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b02d8d12eef3d091e8abd04edd66bc07661dfcaad000c96059425723c2798420/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/acba075203aa617d14da057e0454d43c005ca2f6118ee3dca64504ca26938308/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/cf9af1bc1aaaa291ae86446f7ff0d072a127618cc913198a47afad631c8e5f09/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2ad0e481e12e53e7a56fb44df7ca52c10b3c7f0702dce6b13bea2c008d356561/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0ab8e551803c4eaf44f6245cae291f35cbf89c8efc7d5803b3fa371d9a30ef13/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2544cf74d1040320e457efa8e26e084bdb1ed2c78cb1c72090a0c7bc6c9aaee2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/8388eddaf9e985939c509320c93bffe4463280405517e54e24f3c30cad399a9d/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/0e2ad0801559497cc0f48884232382e269d9d873194e8b680b069fa85f1eef5b/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/efb45aa63f0c4bc2b835304789aa5d648f6ef8d69937240ecbbc67e49eb00999/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/88132d942030d2aacfcffd243e6594860ae4ed02013cf9f51843512476fac318/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6753b68a9f0e5560807dc8a3907f32cfee073aa514af2097c8fe5a46ce2c05d9/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/57e237b24331532d359a04201e6246de466411b1dd2555a22379143aceede7e9/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/b82719f219a21f426d7829c057e0b4ecf14f030e7f6613dd3727da19fdcb7cc6/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/7421384055c08bdad7580d6050f5d8bdf0d99a114537b8a63245182ef4459eb1/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/8049d7c3e8b4090cc53f6fe84120fa3abbdbb87bf3867d55a16eab9b5e09db8c/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/2b9a1fc14fde47462de08a53c8ad3eac7903ebf1c45d65bc0e1a937dfef006c6/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/0731bf485cd4e0c26043d3d6e8861c9d859203d2e86b5cba7de6dab556b0265f/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/7059df5c8a475c2b1e2f0b05aab96c0d7e323e38501fc73717272173960be636/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/cf1c7081a0cebba67c8ea3f8fc0352fff8b6a5e8ffd42baa7306166a252606b4/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/9abb96216a58768fc415283f4e4f18db2c7a981600423044780c9be3d140ebf6/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/9affc3435f1b01fe3aaea5de153345db8e43a459ab2a1fea739db4b24fc3c1d5/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/90652b74e9277139b5b0ec6d7c4b6c83a7ae0c255d6dfb672eb652cf40755566/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/93dcfb8df0f260ab0bf2a7d35be6d299e1c50f3d00357c1e15817e2bfd8fad77/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/9653a786db0014f3a9673a09d809c3ceed631eb27cb2d8ff61ff47ce703dfe66/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/690e6ff926d5b003fcbd80ee34e0ee828bd675b410fded12e0962fffca0bf1d7/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/00dd6e81db142eea3fcab94077e53e72813f4c45d0d73d088285692525c3ec58/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/5e054baaeaa438f7e7a24286071fc38fcd8021ed8ce06c25fa252a663e52120c/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/bc081ad652970105f41d07fd389d046d398efa32ef23f34b44daa2c65299cc66/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/89736e9e2627eec147fff53f86a09605b26f447ee430dc2505fd7e143be8b601/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/b3b4d9a9b2fd3ff2d8e88e257e640fc26630fcab3a247abc4f55ea9ca514ef7e/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/0179e47dc05f1ffde8ebfbb1a282391b05288020a15330e5583b923dcfd2c0e5/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/1e4083298251b1f4a0d9f79b18dfd453b41e46040ea965773f958f9802e7ad47/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/c8b14afdc5365df291b73f0a77d17277e5151d7e4170436621bfe04d06cd9665/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/6a90677d92d0502abbc61a8839c0677547eb0142201ab96eca75fa02fc006efc/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/d5cb62496a2f880027c84594e29c4cfe935e3c3397752a7ac41467de2b4bf8b4/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/e74f9dcb62540928ec5ec351c62ad0b43e470fa15edac080fd97d48a68333af3/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/dd3fee0115190394c9b116149228e55c5fe092c93d9b744bcab0e858b8cb9845/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/af21ce560c0d2393cc6c5466e405a576a5c46c9a0421f88283e309176115bfc7/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/d0a22974b60b852206a28f6b12c8b42eb51c956acbbfe38be65c0ddf833a11aa/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/93052415de219656a6cb3170d9233940d35117c93c92c2f3b0180c6626319a1d/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/529d8de6699bee2b1d3e843e66bb1bd3e8a69efa569406ae4181039318f17497/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/c4646d93e0720a891b8c4479f1c4607084b2ef5afea7c3c3be3eb54cde3acc22/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b65dcb7621b49e2f6d59483aece5114cbb825565df896e81fc889ee3e4ee4c6e/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a119d9c7980ff9c3678a2d37ac5bfcca5b0c8e726cd9d5bb60c91025a8ee8736/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/78b7852ea55c331aeca97e0597aaab4425fc0f634ba9afe19c3b652ac1c43a4b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9f5040865cb3093acc414796a5e2fee8e29aa39c00f9dd91aba74717d0e4b772/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c0659421937a0463dccf31a0f30bbc4fb9bc46eef759f7ada0f01fc4ca927b94/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/614c254d624592c6d1c9fc4eabf106327ea022183838459a95ddaab72465ad90/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c4ce5ada831274cfdf2f3f42750ad26469e8792a70a8db7363ac8625f2235b92/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/1dcde698779d4d57d7a631707f1886d3f52e209a6916fb6ac928e9e82cf936db/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2d7b4cf88d8617afd6916d56a4a57ff108f4c0bcf045d4b69b162b621a244770/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 07:54:20.679461 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.679075 5951 manager.go:217] Machine: {Timestamp:2026-11-03 07:54:20.678587612 +0000 UTC m=+0.060038521 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3219477 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e1af23aec19137c69f8eb9b33df7 SystemUUID:ec23e1af-23ae-c191-37c6-9f8eb9b33df7 BootID:bbe370d0-6677-4e1b-af87-2ed20d1fe61f Filesystems:[{Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ac7aabc75c7738962fd01177f4a6d80c8b4bbbc977281c41f4781a03b04a768/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/30259dd3-c5f2-46b2-a5e1-acdfe2a77416/volumes/kubernetes.io~projected/kube-api-access-lsvqn DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86ebef13cb596d1f4fff7c70ce5f85306c2b71853e1c5191f04cb3f4ce63bef4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c82f546a29940800cdabf99d96ac48b8a75700d5fd50237e97d8b0c5c0fbe21/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~projected/kube-api-access-4nhgc DeviceMajor:0 DeviceMinor:276 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1758c1e33597ef1592c9543c17866220b2eb1c56ec2a2efc7a5175e6c00298e5/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:327 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3a66f1e-1c9c-477d-b174-5763e095ba84/volumes/kubernetes.io~projected/kube-api-access-f8jqk DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/af81fa3b-465f-4608-bfe1-cf082d873a95/volumes/kubernetes.io~projected/kube-api-access-7ndsf DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a678cb0800db5ef33a73fc1cd1f9192c6a3118e25d23bd14816231ed1fbe75c3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24c1fbd52801793dac1e1b6584b578cdb9cd52c8d8663719e75679c4b85017b7/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:298 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/db268603-dbe4-4ca3-8f27-b4e27414f5e1/volumes/kubernetes.io~projected/kube-api-access-n9nfk DeviceMajor:0 DeviceMinor:393 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:270 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fba44899-0b4f-4ae4-86c0-a6e4657c9c73/volumes/kubernetes.io~projected/kube-api-access-lvkjn DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~projected/kube-api-access-plvz6 DeviceMajor:0 DeviceMinor:232 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:325 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae52dafe8adaff80136c125adf73373a0ba640a6b7a229fdbaa9c7ef8f953078/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/volumes/kubernetes.io~projected/kube-api-access-s8p78 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f3a66f1e-1c9c-477d-b174-5763e095ba84/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:392 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:272 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0235634f3212307586016adbca593333183ca89fcfca2b6abef801d74190b49c/userdata/shm DeviceMajor:0 DeviceMinor:396 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15c16b9e77eef1e6654179df586bed1b96776ab7266cf16fed3eae67ba8daa4f/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7615d94-79a8-4c1e-ad82-c3034722c74f/volumes/kubernetes.io~projected/kube-api-access-b2bkl DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fba44899-0b4f-4ae4-86c0-a6e4657c9c73/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:332 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/728dc542-604c-46b2-9313-79f41aaf379e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc7365d92d743cc6f2f922dcd7c261aae0a0064721959963d090899b5d2df809/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5497a21d-0e9d-4191-bf7b-392785b052d0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f7a83c0b90546ffee8c581358f6de48daf57091c18aba3d5069e530770d74ad/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a59b6e092dca03b8ef26611c782f8bfc8112d40811509b3ed67c9c8001173a0b/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d5892f1a5264e7ee08c6787500e0c3bfbf9b33ce26f559b47c39081ab48799c/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:273 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce8c5d5b-8c05-4057-a77d-4a102186c208/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4d8a3e7e2dd0fdeb0fe4b321559af08f06535fdc9e8f8fb2af37bd82b1d9e5d/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/712391ec-a5a3-4824-9219-a1b56b5f214e/volumes/kubernetes.io~projected/kube-api-access-kxz9h DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb3f89e4ae521b45deba6833e403e2df072fa39c19adaae0a85090b554d5875e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:320 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6c125505-d546-4704-9e55-ecb5b867aa4d/volumes/kubernetes.io~projected/kube-api-access-28mqm DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5497a21d-0e9d-4191-bf7b-392785b052d0/volumes/kubernetes.io~projected/kube-api-access-44xgg DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10e03d381d24c47cb6a799361ea2490821d33ae5556d7fc9df24836211d5096f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abfaf52a-6e21-4290-93cf-367a0aa9d749/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:388 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce951427-6f0a-46cc-8ac9-9776395bc3b8/volumes/kubernetes.io~projected/kube-api-access-jwxst DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:331 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:271 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/volumes/kubernetes.io~projected/kube-api-access-5gmjv DeviceMajor:0 DeviceMinor:302 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:324 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:326 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:274 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes/kubernetes.io~projected/kube-api-access-r5wjs DeviceMajor:0 DeviceMinor:334 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63ce7d86-82e4-4e3e-8aa7-0a131032133b/volumes/kubernetes.io~projected/kube-api-access-ss78v DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84e304c5-3d08-4c75-8589-1ee0e67c4ed6/volumes/kubernetes.io~projected/kube-api-access-bjcfs DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2947e2ddd1c8e00c683441530eacde27e028e1b547f6241fa50ca322c1211b14/userdata/shm DeviceMajor:0 DeviceMinor:394 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34/volumes/kubernetes.io~projected/kube-api-access-6vs4s DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e9cb625bf5a97dd7e76f8a39df1b85c414e9cfc8792702f92a04b151dc663cb/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0235634f3212307 MacAddress:12:2d:36:3a:f5:51 Speed:10000 Mtu:8901} {Name:0f7a83c0b90546f MacAddress:0a:9f:99:bc:f1:ba Speed:10000 Mtu:8901} {Name:1758c1e33597ef1 MacAddress:a6:cb:63:c7:37:86 Speed:10000 Mtu:8901} {Name:2947e2ddd1c8e00 MacAddress:02:f4:6b:d1:83:9f Speed:10000 Mtu:8901} {Name:4ac7aabc75c7738 MacAddress:ea:2f:47:de:9a:aa Speed:10000 Mtu:8901} {Name:4dfc6e47475f4b3 MacAddress:1e:6a:9d:8e:83:e1 Speed:10000 Mtu:8901} {Name:6d5892f1a5264e7 MacAddress:8e:a0:c6:c5:54:9e Speed:10000 Mtu:8901} {Name:ae52dafe8adaff8 MacAddress:7e:bf:ec:3a:ad:e5 Speed:10000 Mtu:8901} {Name:b4d8a3e7e2dd0fd MacAddress:b6:f7:2e:20:ce:91 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d3:20:2b:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:66:02:0c:1b:e9 Speed:0 Mtu:8901} {Name:cb3f89e4ae521b4 MacAddress:7e:1a:ae:81:a5:91 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d3:20:2b:49 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:ed:79:64:5d:54 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:2e:1a:79:62:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 07:54:20.679664 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.679465 5951 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 07:54:20.679664 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.679512 5951 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 07:54:20.679864 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.679840 5951 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 07:54:20.679990 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.679865 5951 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-47.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 07:54:20.680032 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.679992 5951 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 07:54:20.680032 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680000 5951 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 07:54:20.680032 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680010 5951 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 07:54:20.680032 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680030 5951 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 07:54:20.680163 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680120 5951 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 07:54:20.680301 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680291 5951 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 07:54:20.680416 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680408 5951 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 07:54:20.680447 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680444 5951 kubelet.go:493] "Attempting to sync node with API server" Nov 03 07:54:20.680504 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680453 5951 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 07:54:20.680504 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680463 5951 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 07:54:20.680504 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680473 5951 kubelet.go:405] "Adding apiserver pod source" Nov 03 07:54:20.680504 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.680481 5951 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 07:54:20.681235 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.681218 5951 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 07:54:20.681235 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.681236 5951 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 07:54:20.681777 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.681757 5951 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 07:54:20.684239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684224 5951 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 07:54:20.684239 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684238 5951 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 07:54:20.684354 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684337 5951 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 07:54:20.684531 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684520 5951 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 07:54:20.684561 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684550 5951 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 07:54:20.684656 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684634 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 07:54:20.684699 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684682 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 07:54:20.684699 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684690 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 07:54:20.684699 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684696 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684701 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684707 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684712 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684717 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684725 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684731 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684745 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684755 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684779 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 07:54:20.684783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.684785 5951 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 07:54:20.688248 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.688221 5951 server.go:1263] "Started kubelet" Nov 03 07:54:20.688351 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.688310 5951 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 07:54:20.690149 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.688435 5951 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 07:54:20.690149 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.688497 5951 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 07:54:20.690149 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.689600 5951 server.go:317] "Adding debug handlers to kubelet server" Nov 03 07:54:20.688796 ip-10-0-143-47 systemd[1]: Started Kubernetes Kubelet. Nov 03 07:54:20.690574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.690180 5951 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 07:54:20.699624 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.699530 5951 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:54:20.700527 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700500 5951 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700682 5951 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700873 5951 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700942 5951 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700958 5951 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700948 5951 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:48:10 +0000 UTC" deadline="2028-08-20 23:34:09.139311652 +0000 UTC" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.700975 5951 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15759h39m48.438342414s" Nov 03 07:54:20.702268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.701003 5951 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703203 5951 factory.go:55] Registering systemd factory Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703229 5951 factory.go:223] Registration of the systemd container factory successfully Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703635 5951 factory.go:153] Registering CRI-O factory Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703678 5951 factory.go:223] Registration of the crio container factory successfully Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703705 5951 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703744 5951 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703767 5951 factory.go:103] Registering Raw factory Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703785 5951 manager.go:1199] Started watching for new ooms in manager Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703761 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c125505-d546-4704-9e55-ecb5b867aa4d" volumeName="kubernetes.io/projected/6c125505-d546-4704-9e55-ecb5b867aa4d-kube-api-access-28mqm" seLinuxMountContext="" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703819 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703839 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703854 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-grpc-tls" seLinuxMountContext="" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703868 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce951427-6f0a-46cc-8ac9-9776395bc3b8" volumeName="kubernetes.io/projected/ce951427-6f0a-46cc-8ac9-9776395bc3b8-kube-api-access-jwxst" seLinuxMountContext="" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703882 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" volumeName="kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 07:54:20.703893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703900 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" volumeName="kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703914 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" volumeName="kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-binary-copy" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703937 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30259dd3-c5f2-46b2-a5e1-acdfe2a77416" volumeName="kubernetes.io/configmap/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-serviceca" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703952 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5497a21d-0e9d-4191-bf7b-392785b052d0" volumeName="kubernetes.io/configmap/5497a21d-0e9d-4191-bf7b-392785b052d0-config-volume" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703967 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c125505-d546-4704-9e55-ecb5b867aa4d" volumeName="kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-config" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.703999 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/empty-dir/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-tmp" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704017 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8c5d5b-8c05-4057-a77d-4a102186c208" volumeName="kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704033 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" volumeName="kubernetes.io/projected/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-kube-api-access-6vs4s" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704056 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c125505-d546-4704-9e55-ecb5b867aa4d" volumeName="kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-env-overrides" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704072 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 07:54:20.704610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704088 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704760 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704794 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af81fa3b-465f-4608-bfe1-cf082d873a95" volumeName="kubernetes.io/projected/af81fa3b-465f-4608-bfe1-cf082d873a95-kube-api-access-7ndsf" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704809 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704831 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba" volumeName="kubernetes.io/projected/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-kube-api-access-s8p78" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704843 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c125505-d546-4704-9e55-ecb5b867aa4d" volumeName="kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-script-lib" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.704860 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" volumeName="kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-metrics-client-ca" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705009 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705025 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705044 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 07:54:20.705067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705057 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705079 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce951427-6f0a-46cc-8ac9-9776395bc3b8" volumeName="kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-daemon-config" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705106 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728dc542-604c-46b2-9313-79f41aaf379e" volumeName="kubernetes.io/configmap/728dc542-604c-46b2-9313-79f41aaf379e-konnectivity-ca" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705119 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8c5d5b-8c05-4057-a77d-4a102186c208" volumeName="kubernetes.io/empty-dir/ce8c5d5b-8c05-4057-a77d-4a102186c208-volume-directive-shadow" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705134 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba44899-0b4f-4ae4-86c0-a6e4657c9c73" volumeName="kubernetes.io/configmap/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705153 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba44899-0b4f-4ae4-86c0-a6e4657c9c73" volumeName="kubernetes.io/empty-dir/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-data-volume" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705165 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5497a21d-0e9d-4191-bf7b-392785b052d0" volumeName="kubernetes.io/projected/5497a21d-0e9d-4191-bf7b-392785b052d0-kube-api-access-44xgg" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705184 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ce7d86-82e4-4e3e-8aa7-0a131032133b" volumeName="kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-tmp" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705222 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705234 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705253 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba44899-0b4f-4ae4-86c0-a6e4657c9c73" volumeName="kubernetes.io/secret/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705266 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b" volumeName="kubernetes.io/projected/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-kube-api-access-5gmjv" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705285 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30259dd3-c5f2-46b2-a5e1-acdfe2a77416" volumeName="kubernetes.io/projected/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-kube-api-access-lsvqn" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705297 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" volumeName="kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705310 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" volumeName="kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-tls" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705327 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705340 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705436 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ce7d86-82e4-4e3e-8aa7-0a131032133b" volumeName="kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-tuned" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705449 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705462 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5497a21d-0e9d-4191-bf7b-392785b052d0" volumeName="kubernetes.io/empty-dir/5497a21d-0e9d-4191-bf7b-392785b052d0-tmp-dir" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705479 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" volumeName="kubernetes.io/empty-dir/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-textfile" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705495 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705513 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5497a21d-0e9d-4191-bf7b-392785b052d0" volumeName="kubernetes.io/secret/5497a21d-0e9d-4191-bf7b-392785b052d0-metrics-tls" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705525 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 07:54:20.705541 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705542 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abfaf52a-6e21-4290-93cf-367a0aa9d749" volumeName="kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705560 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8c5d5b-8c05-4057-a77d-4a102186c208" volumeName="kubernetes.io/projected/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-api-access-plvz6" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705573 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712391ec-a5a3-4824-9219-a1b56b5f214e" volumeName="kubernetes.io/empty-dir/712391ec-a5a3-4824-9219-a1b56b5f214e-tmp-dir" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705591 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712391ec-a5a3-4824-9219-a1b56b5f214e" volumeName="kubernetes.io/projected/712391ec-a5a3-4824-9219-a1b56b5f214e-kube-api-access-kxz9h" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705604 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705617 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8c5d5b-8c05-4057-a77d-4a102186c208" volumeName="kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705694 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce951427-6f0a-46cc-8ac9-9776395bc3b8" volumeName="kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cni-binary-copy" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705708 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3a66f1e-1c9c-477d-b174-5763e095ba84" volumeName="kubernetes.io/projected/f3a66f1e-1c9c-477d-b174-5763e095ba84-kube-api-access-f8jqk" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705725 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3a66f1e-1c9c-477d-b174-5763e095ba84" volumeName="kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705738 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7615d94-79a8-4c1e-ad82-c3034722c74f" volumeName="kubernetes.io/projected/f7615d94-79a8-4c1e-ad82-c3034722c74f-kube-api-access-b2bkl" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705751 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba44899-0b4f-4ae4-86c0-a6e4657c9c73" volumeName="kubernetes.io/projected/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-api-access-lvkjn" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705768 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" volumeName="kubernetes.io/projected/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-kube-api-access-bjcfs" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705782 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705798 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705813 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/projected/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-kube-api-access-4nhgc" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705825 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db268603-dbe4-4ca3-8f27-b4e27414f5e1" volumeName="kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705846 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c125505-d546-4704-9e55-ecb5b867aa4d" volumeName="kubernetes.io/secret/6c125505-d546-4704-9e55-ecb5b867aa4d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705860 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705879 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705934 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/configmap/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-metrics-client-ca" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705946 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8c5d5b-8c05-4057-a77d-4a102186c208" volumeName="kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-metrics-client-ca" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705963 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b" volumeName="kubernetes.io/secret/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705977 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" volumeName="kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.705994 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902ad59c-068f-42e8-aa8d-d1658a31e3bf" volumeName="kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706005 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706018 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8c5d5b-8c05-4057-a77d-4a102186c208" volumeName="kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706035 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba" volumeName="kubernetes.io/secret/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-cert" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706047 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63ce7d86-82e4-4e3e-8aa7-0a131032133b" volumeName="kubernetes.io/projected/63ce7d86-82e4-4e3e-8aa7-0a131032133b-kube-api-access-ss78v" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706065 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="728dc542-604c-46b2-9313-79f41aaf379e" volumeName="kubernetes.io/secret/728dc542-604c-46b2-9313-79f41aaf379e-agent-certs" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706077 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af81fa3b-465f-4608-bfe1-cf082d873a95" volumeName="kubernetes.io/configmap/af81fa3b-465f-4608-bfe1-cf082d873a95-iptables-alerter-script" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706093 5951 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff1dba0-28ed-4e7f-ae34-9d30f7b881a1" volumeName="kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706107 5951 reconstruct.go:97] "Volume reconstruction finished" Nov 03 07:54:20.706557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706115 5951 reconciler.go:29] "Reconciler: start to sync state" Nov 03 07:54:20.708030 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.706603 5951 manager.go:319] Starting recovery of all containers Nov 03 07:54:20.710014 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.709340 5951 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 07:54:20.715323 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.715280 5951 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 07:54:20.728485 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.728446 5951 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 07:54:20.728485 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.728484 5951 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 07:54:20.728722 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.728507 5951 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 07:54:20.728722 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:20.728550 5951 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 07:54:20.731546 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.731516 5951 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 07:54:20.736093 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.736066 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:54:20.737513 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.737490 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:54:20.739129 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.739114 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:54:20.739405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.739383 5951 generic.go:356] "Generic (PLEG): container finished" podID="6c125505-d546-4704-9e55-ecb5b867aa4d" containerID="6ac15cff00f326810ca67ca0999755dde71bdf870a3659a005e6682d0b7b7495" exitCode=1 Nov 03 07:54:20.749839 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.749818 5951 generic.go:356] "Generic (PLEG): container finished" podID="84e304c5-3d08-4c75-8589-1ee0e67c4ed6" containerID="9599b122983c8b1892b30766e4bd5768ecbaab714365ef328b6dd989328a9f5e" exitCode=0 Nov 03 07:54:20.755388 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.755368 5951 generic.go:356] "Generic (PLEG): container finished" podID="f590e0382a000936d43fee8841977af7" containerID="ec75a9d8e51ec75edb764b053f0f2f2c6b07a90d2c0a2ac62188c934775b45b7" exitCode=0 Nov 03 07:54:20.758910 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.758890 5951 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="1695981a9ea7bcafce00c302f656b518c40bc81563531bc2e47ea41f7f11a442" exitCode=0 Nov 03 07:54:20.758910 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.758909 5951 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="3c761dbf2bbf2c394b018b0d16edcd807fc994b19727145f0e00e7696a1f78a2" exitCode=0 Nov 03 07:54:20.759049 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.758917 5951 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="6af5eab8b0fe3c90617543d77d3aecb145874dcd83913315c95efba5a60f1b03" exitCode=0 Nov 03 07:54:20.759049 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.758926 5951 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="7a60eb848962a4b4b7140ff206a71a56de1443115fe249f379d302efb2be516e" exitCode=0 Nov 03 07:54:20.759049 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.758934 5951 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="8e15b08b88e3e4e8001e1d01e60247c4593308ab04fdc3ce266eea0c8b693db3" exitCode=0 Nov 03 07:54:20.759049 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.758942 5951 generic.go:356] "Generic (PLEG): container finished" podID="13b35f75-7b9a-4a00-a775-8f2ae7cc9a34" containerID="ee611d01fe56f57b31dd71e25812c824d5846be8470834fc25bc99377b62fe1a" exitCode=0 Nov 03 07:54:20.779988 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.779964 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" exitCode=0 Nov 03 07:54:20.806128 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.806108 5951 manager.go:324] Recovery completed Nov 03 07:54:20.828220 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828167 5951 cpu_manager.go:226] "Starting" policy="none" Nov 03 07:54:20.828220 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828182 5951 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 07:54:20.828220 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828198 5951 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 07:54:20.828361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828314 5951 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 07:54:20.828361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828325 5951 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 07:54:20.828361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828348 5951 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 07:54:20.828361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828355 5951 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 07:54:20.828361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828360 5951 policy_none.go:50] "Start" Nov 03 07:54:20.828489 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828369 5951 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 07:54:20.828489 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828379 5951 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 07:54:20.828542 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828518 5951 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 07:54:20.828542 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828530 5951 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 07:54:20.828542 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.828536 5951 policy_none.go:44] "Start" Nov 03 07:54:20.828762 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:20.828680 5951 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 07:54:20.834073 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.834043 5951 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 07:54:20.837349 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.837336 5951 manager.go:341] "Starting Device Plugin manager" Nov 03 07:54:20.837427 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:20.837359 5951 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 07:54:20.837427 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.837368 5951 server.go:87] "Starting device plugin registration server" Nov 03 07:54:20.838365 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838350 5951 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 07:54:20.838429 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838366 5951 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 07:54:20.838499 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838472 5951 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 07:54:20.838629 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838589 5951 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 07:54:20.838744 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838635 5951 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 07:54:20.838744 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838659 5951 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 07:54:20.838839 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.838695 5951 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T07:54:20.838624339Z","UUID":"fa39c059-dded-4229-947b-34f0bfc389f1","Handler":null,"Name":"","Endpoint":""} Nov 03 07:54:20.839977 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.839960 5951 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 07:54:20.840059 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.839992 5951 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 07:54:20.840240 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:20.840220 5951 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 07:54:20.939299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.939266 5951 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 07:54:20.942815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.942798 5951 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 07:54:20.942887 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.942829 5951 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 07:54:20.942887 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.942839 5951 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 07:54:20.942887 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.942878 5951 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:20.956423 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.956399 5951 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:20.956530 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:20.956465 5951 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.029495 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.029418 5951 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal"] Nov 03 07:54:21.030119 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030103 5951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.030207 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030161 5951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.030207 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030127 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"f590e0382a000936d43fee8841977af7","Type":"ContainerStarted","Data":"e55369473eae6ae134090499d3d5e06198fa8c936779b9b9fe9fafa58eee2465"} Nov 03 07:54:21.030308 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030205 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"f590e0382a000936d43fee8841977af7","Type":"ContainerDied","Data":"ec75a9d8e51ec75edb764b053f0f2f2c6b07a90d2c0a2ac62188c934775b45b7"} Nov 03 07:54:21.030308 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030228 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"f590e0382a000936d43fee8841977af7","Type":"ContainerStarted","Data":"588e24d12c8c0dec3054114b0c9a5c04b0f41cc35c7c6dea53f62c82df385fb8"} Nov 03 07:54:21.030308 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030267 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"57afb19d8731b471ac7e459ac8d782fe","Type":"ContainerStarted","Data":"5371d854841dfdf78511ea866f68d6a57d654b870c4decfc8412c6f46b6be4a7"} Nov 03 07:54:21.030308 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.030289 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" event={"ID":"57afb19d8731b471ac7e459ac8d782fe","Type":"ContainerStarted","Data":"9967331cf997689d0e7691bc6246fcbb000fa2a326a008a1d1d71733151b29ff"} Nov 03 07:54:21.041534 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.041514 5951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 07:54:21.041587 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:21.041562 5951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.041896 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.041883 5951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 07:54:21.041935 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:21.041918 5951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.111313 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.111240 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.111313 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.111272 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.111313 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.111291 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57afb19d8731b471ac7e459ac8d782fe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"57afb19d8731b471ac7e459ac8d782fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.211915 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.211856 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.212000 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.211928 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.212000 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.211957 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57afb19d8731b471ac7e459ac8d782fe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"57afb19d8731b471ac7e459ac8d782fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.212000 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.211966 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.212000 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.211994 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57afb19d8731b471ac7e459ac8d782fe-config\") pod \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"57afb19d8731b471ac7e459ac8d782fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.212129 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.212039 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f590e0382a000936d43fee8841977af7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" (UID: \"f590e0382a000936d43fee8841977af7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.681077 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.681043 5951 apiserver.go:52] "Watching apiserver" Nov 03 07:54:21.686818 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.686800 5951 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 07:54:21.687390 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.687369 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-llmvz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww","openshift-monitoring/node-exporter-vtg6x","openshift-image-registry/node-ca-skwds","openshift-ingress-canary/ingress-canary-j4bwh","openshift-network-diagnostics/network-check-target-brfqb","openshift-dns/dns-default-zqtl5","openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh","openshift-multus/multus-lnzwz","openshift-cluster-node-tuning-operator/tuned-5gvgx","openshift-multus/network-metrics-daemon-d7m5s","openshift-network-operator/iptables-alerter-w9w62","openshift-ovn-kubernetes/ovnkube-node-ww6bv","openshift-multus/multus-additional-cni-plugins-vfclf","openshift-dns/node-resolver-nbxj7","openshift-insights/insights-runtime-extractor-pp56g","openshift-monitoring/thanos-querier-f889895df-lctqt","openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj","kube-system/global-pull-secret-syncer-hqxvr","openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:21.692006 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.691980 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2mmmx\"" Nov 03 07:54:21.692393 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.692366 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 07:54:21.693341 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.693298 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 07:54:21.693445 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.693339 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.693873 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.693850 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 07:54:21.693991 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.693975 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmmvq\"" Nov 03 07:54:21.694038 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.694013 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.694148 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.694130 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 07:54:21.694724 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.694706 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 07:54:21.694853 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.694829 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwf4\"" Nov 03 07:54:21.694969 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.694902 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.695026 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.694986 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 07:54:21.695221 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695200 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.695323 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695303 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.695375 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695365 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 07:54:21.695422 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695380 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpwxk\"" Nov 03 07:54:21.695422 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695413 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.695738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695720 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 07:54:21.695802 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695778 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\"" Nov 03 07:54:21.695857 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695830 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjbhz\"" Nov 03 07:54:21.695905 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695867 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\"" Nov 03 07:54:21.696078 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696058 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.696132 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696100 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-thqm7\"" Nov 03 07:54:21.696198 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696179 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.696247 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696196 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 07:54:21.696247 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.695728 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9glt\"" Nov 03 07:54:21.696364 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696344 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.696417 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696372 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.696469 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696421 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.696469 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696064 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 07:54:21.696557 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696526 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.696602 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696570 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 07:54:21.696680 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696634 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 07:54:21.696736 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696180 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 07:54:21.696736 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696715 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4xzk\"" Nov 03 07:54:21.696824 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696789 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dc6md\"" Nov 03 07:54:21.696882 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696866 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2t62\"" Nov 03 07:54:21.696985 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696968 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 07:54:21.697045 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696984 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.697045 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.696968 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.697045 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.697039 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.697161 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.697048 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 07:54:21.697161 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.697049 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.697316 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.697300 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\"" Nov 03 07:54:21.697911 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.697891 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 07:54:21.697982 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.697956 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 07:54:21.698978 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.698962 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 07:54:21.699346 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.699333 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.699537 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.699521 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.699897 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.699879 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.700120 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700101 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.700391 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700374 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5spf6\"" Nov 03 07:54:21.700479 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700403 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ghck6\"" Nov 03 07:54:21.700479 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700458 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 07:54:21.700582 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700500 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 07:54:21.700634 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700590 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 07:54:21.700634 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700603 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8t4tj\"" Nov 03 07:54:21.700771 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700712 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 07:54:21.700771 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700727 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 07:54:21.700852 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700814 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 07:54:21.700919 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700903 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 07:54:21.700978 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.700911 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 07:54:21.701029 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701007 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 07:54:21.701121 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701107 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 07:54:21.701321 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701305 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8pp7x\"" Nov 03 07:54:21.701384 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701350 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 07:54:21.701789 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701767 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 07:54:21.701859 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701808 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 07:54:21.701859 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701822 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 07:54:21.701987 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.701971 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 07:54:21.702106 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.702093 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-17ekhfl0cqidc\"" Nov 03 07:54:21.702164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.702103 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6cpzp\"" Nov 03 07:54:21.702503 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.702485 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m644k\"" Nov 03 07:54:21.702556 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.702544 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 07:54:21.702629 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.702619 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 07:54:21.704019 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704007 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 07:54:21.704202 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704184 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 07:54:21.704258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704217 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8f9uviit3bo6o\"" Nov 03 07:54:21.704258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704219 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 07:54:21.704357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704265 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 07:54:21.704357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704303 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 07:54:21.704357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704315 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 07:54:21.704357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704329 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 07:54:21.704357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704343 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 07:54:21.704357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704356 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 07:54:21.704523 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704497 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 07:54:21.704555 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704532 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 07:54:21.704700 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704665 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 07:54:21.704700 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704688 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bxcxf\"" Nov 03 07:54:21.704867 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.704781 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 07:54:21.706172 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.706158 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 07:54:21.706279 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.706266 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 07:54:21.714752 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.714730 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-system-cni-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.714842 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.714755 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-root\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.714842 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.714778 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.714842 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.714796 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715052 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715040 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-multus-certs\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.715052 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715041 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.715117 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715068 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:21.715117 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715086 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:21.715222 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715125 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715222 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715156 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-run\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.715222 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715203 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715222 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715210 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-kubelet-config\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.715398 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715244 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-env-overrides\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.715398 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715279 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhgc\" (UniqueName: \"kubernetes.io/projected/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-kube-api-access-4nhgc\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.715398 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715307 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3a66f1e-1c9c-477d-b174-5763e095ba84-metrics-certs\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:21.715398 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715340 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715398 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715343 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715398 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715385 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715414 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-serviceca\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715507 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af81fa3b-465f-4608-bfe1-cf082d873a95-host-slash\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715510 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715421 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-env-overrides\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715548 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715577 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-daemon-config\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715623 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-bin\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.715687 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715681 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-wtmp\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715711 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vs4s\" (UniqueName: \"kubernetes.io/projected/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-kube-api-access-6vs4s\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715741 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5497a21d-0e9d-4191-bf7b-392785b052d0-config-volume\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715760 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/abfaf52a-6e21-4290-93cf-367a0aa9d749-original-pull-secret\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715777 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-grpc-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715777 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-daemon-config\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715799 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-node-log\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.715914 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716024 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716033 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5497a21d-0e9d-4191-bf7b-392785b052d0-config-volume\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716102 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xgg\" (UniqueName: \"kubernetes.io/projected/5497a21d-0e9d-4191-bf7b-392785b052d0-kube-api-access-44xgg\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716109 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-serviceca\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.716141 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716134 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-grpc-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716180 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716206 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-tls\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716233 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716269 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-log-socket\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716375 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716420 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-var-lib-kubelet\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716464 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716492 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-textfile\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716564 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-textfile\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716579 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcfs\" (UniqueName: \"kubernetes.io/projected/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-kube-api-access-bjcfs\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716565 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716592 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716610 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af81fa3b-465f-4608-bfe1-cf082d873a95-iptables-alerter-script\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716636 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.716765 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716661 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-multus\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716794 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716823 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af81fa3b-465f-4608-bfe1-cf082d873a95-iptables-alerter-script\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716854 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716875 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-device-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716908 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-systemd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.716944 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717013 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717041 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717022 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717061 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-bin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717098 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c125505-d546-4704-9e55-ecb5b867aa4d-ovn-node-metrics-cert\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717174 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717182 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-script-lib\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717227 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mqm\" (UniqueName: \"kubernetes.io/projected/6c125505-d546-4704-9e55-ecb5b867aa4d-kube-api-access-28mqm\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717246 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717250 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5497a21d-0e9d-4191-bf7b-392785b052d0-tmp-dir\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717260 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717282 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717329 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5497a21d-0e9d-4191-bf7b-392785b052d0-tmp-dir\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717359 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c125505-d546-4704-9e55-ecb5b867aa4d-ovn-node-metrics-cert\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717361 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-lib-modules\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717396 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-tuned\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717411 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-script-lib\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717439 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-netns\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717455 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-tuned\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717467 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxst\" (UniqueName: \"kubernetes.io/projected/ce951427-6f0a-46cc-8ac9-9776395bc3b8-kube-api-access-jwxst\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717507 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-socket-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717515 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717525 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717540 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-k8s-cni-cncf-io\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.717551 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717566 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-metrics-client-ca\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717786 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-metrics-client-ca\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.717805 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718056 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-kubelet\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718077 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-systemd\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718094 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718111 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718126 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718140 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkjn\" (UniqueName: \"kubernetes.io/projected/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-api-access-lvkjn\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718156 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718170 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-metrics-client-ca\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718186 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718215 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718230 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-modprobe-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718243 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718257 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-netd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718270 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718301 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718319 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-socket-dir-parent\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718333 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-host\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718347 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-data-volume\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718363 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718378 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/728dc542-604c-46b2-9313-79f41aaf379e-agent-certs\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718392 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvqn\" (UniqueName: \"kubernetes.io/projected/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-kube-api-access-lsvqn\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718406 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cnibin\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718420 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-kubernetes\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718435 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-cert\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718449 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718474 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712391ec-a5a3-4824-9219-a1b56b5f214e-tmp-dir\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718666 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718720 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxz9h\" (UniqueName: \"kubernetes.io/projected/712391ec-a5a3-4824-9219-a1b56b5f214e-kube-api-access-kxz9h\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.719108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.718749 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-host\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719245 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719254 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-etc-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719246 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719293 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719309 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719327 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jqk\" (UniqueName: \"kubernetes.io/projected/f3a66f1e-1c9c-477d-b174-5763e095ba84-kube-api-access-f8jqk\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719332 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-cert\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719333 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719382 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-data-volume\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719517 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719548 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-metrics-client-ca\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719732 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719850 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719887 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-netns\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719893 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719915 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-etc-kubernetes\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719948 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce8c5d5b-8c05-4057-a77d-4a102186c208-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.719978 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-var-lib-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720079 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720103 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-system-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720126 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/728dc542-604c-46b2-9313-79f41aaf379e-agent-certs\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720408 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-tmp\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720441 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720478 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvz6\" (UniqueName: \"kubernetes.io/projected/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-api-access-plvz6\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720501 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ce8c5d5b-8c05-4057-a77d-4a102186c208-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720514 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-systemd-units\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720551 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-tmp\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720693 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720721 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-hostroot\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720754 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720785 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-etc-selinux\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.720815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.720819 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bkl\" (UniqueName: \"kubernetes.io/projected/f7615d94-79a8-4c1e-ad82-c3034722c74f-kube-api-access-b2bkl\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721261 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/db268603-dbe4-4ca3-8f27-b4e27414f5e1-kube-api-access-n9nfk\") pod \"network-check-target-brfqb\" (UID: \"db268603-dbe4-4ca3-8f27-b4e27414f5e1\") " pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721379 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-config\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721418 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721430 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vs4s\" (UniqueName: \"kubernetes.io/projected/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-kube-api-access-6vs4s\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721455 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721525 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-conf-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721570 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721574 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/728dc542-604c-46b2-9313-79f41aaf379e-konnectivity-ca\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721659 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721703 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysconfig\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721777 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63ce7d86-82e4-4e3e-8aa7-0a131032133b-tmp\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721824 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/728dc542-604c-46b2-9313-79f41aaf379e-konnectivity-ca\") pod \"konnectivity-agent-llmvz\" (UID: \"728dc542-604c-46b2-9313-79f41aaf379e\") " pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721826 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721866 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.721906 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-conf\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722056 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xgg\" (UniqueName: \"kubernetes.io/projected/5497a21d-0e9d-4191-bf7b-392785b052d0-kube-api-access-44xgg\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722071 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722110 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcfs\" (UniqueName: \"kubernetes.io/projected/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-kube-api-access-bjcfs\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722276 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-crio-socket\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722305 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-sys-fs\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722342 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722380 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cni-binary-copy\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.722425 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722427 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss78v\" (UniqueName: \"kubernetes.io/projected/63ce7d86-82e4-4e3e-8aa7-0a131032133b-kube-api-access-ss78v\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722457 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-dbus\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722462 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c125505-d546-4704-9e55-ecb5b867aa4d-ovnkube-config\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722487 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-sys\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722523 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722522 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5497a21d-0e9d-4191-bf7b-392785b052d0-metrics-tls\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722586 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722623 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cnibin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722730 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722804 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-tmp\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722464 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhgc\" (UniqueName: \"kubernetes.io/projected/aff1dba0-28ed-4e7f-ae34-9d30f7b881a1-kube-api-access-4nhgc\") pod \"thanos-querier-f889895df-lctqt\" (UID: \"aff1dba0-28ed-4e7f-ae34-9d30f7b881a1\") " pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722858 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.722939 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxst\" (UniqueName: \"kubernetes.io/projected/ce951427-6f0a-46cc-8ac9-9776395bc3b8-kube-api-access-jwxst\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723011 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723054 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-ovn\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723096 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723236 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723299 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723307 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723335 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723348 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723353 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723398 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-sys\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723434 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-registration-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723481 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-tls\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723539 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmjv\" (UniqueName: \"kubernetes.io/projected/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-kube-api-access-5gmjv\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723567 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-kubelet\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723564 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mqm\" (UniqueName: \"kubernetes.io/projected/6c125505-d546-4704-9e55-ecb5b867aa4d-kube-api-access-28mqm\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.723552 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723591 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712391ec-a5a3-4824-9219-a1b56b5f214e-tmp-dir\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723608 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5497a21d-0e9d-4191-bf7b-392785b052d0-metrics-tls\") pod \"dns-default-zqtl5\" (UID: \"5497a21d-0e9d-4191-bf7b-392785b052d0\") " pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723705 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.723786 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cni-binary-copy\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724023 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-tls\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724093 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-slash\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724161 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndsf\" (UniqueName: \"kubernetes.io/projected/af81fa3b-465f-4608-bfe1-cf082d873a95-kube-api-access-7ndsf\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724218 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8p78\" (UniqueName: \"kubernetes.io/projected/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-kube-api-access-s8p78\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724315 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jqk\" (UniqueName: \"kubernetes.io/projected/f3a66f1e-1c9c-477d-b174-5763e095ba84-kube-api-access-f8jqk\") pod \"network-metrics-daemon-d7m5s\" (UID: \"f3a66f1e-1c9c-477d-b174-5763e095ba84\") " pod="openshift-multus/network-metrics-daemon-d7m5s" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724573 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724612 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712391ec-a5a3-4824-9219-a1b56b5f214e-hosts-file\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724692 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkjn\" (UniqueName: \"kubernetes.io/projected/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-api-access-lvkjn\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724766 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvqn\" (UniqueName: \"kubernetes.io/projected/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-kube-api-access-lsvqn\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724766 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxz9h\" (UniqueName: \"kubernetes.io/projected/712391ec-a5a3-4824-9219-a1b56b5f214e-kube-api-access-kxz9h\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.725153 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.724827 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.729135 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.729110 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:21.730361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.730332 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvz6\" (UniqueName: \"kubernetes.io/projected/ce8c5d5b-8c05-4057-a77d-4a102186c208-kube-api-access-plvz6\") pod \"kube-state-metrics-5dc8d7d89b-c6pqj\" (UID: \"ce8c5d5b-8c05-4057-a77d-4a102186c208\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c6pqj" Nov 03 07:54:21.730841 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.730815 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bkl\" (UniqueName: \"kubernetes.io/projected/f7615d94-79a8-4c1e-ad82-c3034722c74f-kube-api-access-b2bkl\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.730940 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.730915 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8p78\" (UniqueName: \"kubernetes.io/projected/e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba-kube-api-access-s8p78\") pod \"ingress-canary-j4bwh\" (UID: \"e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba\") " pod="openshift-ingress-canary/ingress-canary-j4bwh" Nov 03 07:54:21.731198 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.731178 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndsf\" (UniqueName: \"kubernetes.io/projected/af81fa3b-465f-4608-bfe1-cf082d873a95-kube-api-access-7ndsf\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.731252 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.731213 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmjv\" (UniqueName: \"kubernetes.io/projected/2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b-kube-api-access-5gmjv\") pod \"monitoring-plugin-77f4f76594-4h9gh\" (UID: \"2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:54:21.731350 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.731334 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss78v\" (UniqueName: \"kubernetes.io/projected/63ce7d86-82e4-4e3e-8aa7-0a131032133b-kube-api-access-ss78v\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.788211 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.788181 5951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.788374 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.788252 5951 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.798070 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.798045 5951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 07:54:21.798170 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:21.798099 5951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.798549 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.798530 5951 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 07:54:21.798671 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:21.798573 5951 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-47.us-east-2.compute.internal" Nov 03 07:54:21.801831 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.801815 5951 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 07:54:21.825458 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825436 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-dbus\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825462 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-sys\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825479 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cnibin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825493 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-ovn\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825508 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825522 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825535 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-sys\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.825559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825548 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-registration-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825563 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-sys\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825583 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-dbus\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825571 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-kubelet\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825586 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-ovn\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825631 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-registration-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825633 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-sys\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825662 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-slash\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825699 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825703 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-cnibin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825709 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825726 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-slash\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825738 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-kubelet\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825742 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712391ec-a5a3-4824-9219-a1b56b5f214e-hosts-file\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825778 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-system-cni-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825786 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/712391ec-a5a3-4824-9219-a1b56b5f214e-hosts-file\") pod \"node-resolver-nbxj7\" (UID: \"712391ec-a5a3-4824-9219-a1b56b5f214e\") " pod="openshift-dns/node-resolver-nbxj7" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825822 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-system-cni-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825835 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-root\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825861 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-root\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.825886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825892 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-multus-certs\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825913 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-run\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825928 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-kubelet-config\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825931 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-multus-certs\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825944 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af81fa3b-465f-4608-bfe1-cf082d873a95-host-slash\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825966 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-bin\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825968 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-run\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825981 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-wtmp\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825987 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/abfaf52a-6e21-4290-93cf-367a0aa9d749-kubelet-config\") pod \"global-pull-secret-syncer-hqxvr\" (UID: \"abfaf52a-6e21-4290-93cf-367a0aa9d749\") " pod="kube-system/global-pull-secret-syncer-hqxvr" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826007 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af81fa3b-465f-4608-bfe1-cf082d873a95-host-slash\") pod \"iptables-alerter-w9w62\" (UID: \"af81fa3b-465f-4608-bfe1-cf082d873a95\") " pod="openshift-network-operator/iptables-alerter-w9w62" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.825998 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-node-log\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826025 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-bin\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826032 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-node-log\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826046 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-log-socket\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826054 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84e304c5-3d08-4c75-8589-1ee0e67c4ed6-node-exporter-wtmp\") pod \"node-exporter-vtg6x\" (UID: \"84e304c5-3d08-4c75-8589-1ee0e67c4ed6\") " pod="openshift-monitoring/node-exporter-vtg6x" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826067 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-var-lib-kubelet\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826100 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-multus\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826125 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826135 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-var-lib-kubelet\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826147 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-log-socket\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826161 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-multus\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826148 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-device-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826180 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-device-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826186 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-systemd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826198 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826210 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-systemd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826237 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-bin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826266 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-lib-modules\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826291 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-netns\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826320 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-socket-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826328 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-cni-bin\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826344 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-k8s-cni-cncf-io\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826365 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-netns\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826368 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-kubelet\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826395 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-systemd\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826384 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-socket-dir\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826407 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-lib-modules\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826435 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-k8s-cni-cncf-io\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826444 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-systemd\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826456 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-var-lib-kubelet\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826470 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-modprobe-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826505 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826530 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-netd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826559 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826537 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-modprobe-d\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826590 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-socket-dir-parent\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826554 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-run-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826590 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-cni-netd\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826611 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-host\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826627 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826629 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-socket-dir-parent\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826669 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cnibin\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826678 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-host\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826714 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b35f75-7b9a-4a00-a775-8f2ae7cc9a34-cnibin\") pod \"multus-additional-cni-plugins-vfclf\" (UID: \"13b35f75-7b9a-4a00-a775-8f2ae7cc9a34\") " pod="openshift-multus/multus-additional-cni-plugins-vfclf" Nov 03 07:54:21.826715 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826736 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-kubernetes\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826755 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-host\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826775 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-etc-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826800 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-netns\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826837 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-etc-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826836 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30259dd3-c5f2-46b2-a5e1-acdfe2a77416-host\") pod \"node-ca-skwds\" (UID: \"30259dd3-c5f2-46b2-a5e1-acdfe2a77416\") " pod="openshift-image-registry/node-ca-skwds" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826876 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-etc-kubernetes\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826856 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-host-run-netns\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826866 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-kubernetes\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826905 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-var-lib-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826932 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-var-lib-openvswitch\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.826967 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827000 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827073 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-system-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827082 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-etc-kubernetes\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827093 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-systemd-units\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827109 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-hostroot\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827128 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-etc-selinux\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827154 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-systemd-units\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827181 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-etc-selinux\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827179 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-hostroot\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827200 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-conf-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827232 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysconfig\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827200 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-system-cni-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827247 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951427-6f0a-46cc-8ac9-9776395bc3b8-multus-conf-dir\") pod \"multus-lnzwz\" (UID: \"ce951427-6f0a-46cc-8ac9-9776395bc3b8\") " pod="openshift-multus/multus-lnzwz" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827277 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysconfig\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827279 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827306 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-conf\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827310 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c125505-d546-4704-9e55-ecb5b867aa4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ww6bv\" (UID: \"6c125505-d546-4704-9e55-ecb5b867aa4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827329 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-crio-socket\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827355 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-sys-fs\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827413 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63ce7d86-82e4-4e3e-8aa7-0a131032133b-etc-sysctl-conf\") pod \"tuned-5gvgx\" (UID: \"63ce7d86-82e4-4e3e-8aa7-0a131032133b\") " pod="openshift-cluster-node-tuning-operator/tuned-5gvgx" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827426 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fba44899-0b4f-4ae4-86c0-a6e4657c9c73-crio-socket\") pod \"insights-runtime-extractor-pp56g\" (UID: \"fba44899-0b4f-4ae4-86c0-a6e4657c9c73\") " pod="openshift-insights/insights-runtime-extractor-pp56g" Nov 03 07:54:21.828366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:21.827432 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7615d94-79a8-4c1e-ad82-c3034722c74f-sys-fs\") pod \"aws-ebs-csi-driver-node-258ww\" (UID: \"f7615d94-79a8-4c1e-ad82-c3034722c74f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-258ww" Nov 03 07:54:23.234426 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:23.234397 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:23.789366 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:23.789336 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:24.755986 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:24.755961 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f889895df-lctqt" Nov 03 07:54:24.849070 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:24.849045 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zqtl5" Nov 03 07:54:25.463854 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:25.463822 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4h9gh" Nov 03 07:54:25.465479 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:25.465461 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:25.465655 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:25.465633 5951 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 07:54:27.150804 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:27.150771 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-llmvz" Nov 03 07:54:27.847842 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:27.847813 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:54:31.764284 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:31.764210 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ww6bv" Nov 03 07:54:31.952448 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:31.952415 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:54:40.269225 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269190 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:40.269738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269579 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy" containerID="cri-o://e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" gracePeriod=600 Nov 03 07:54:40.269738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269602 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="thanos-sidecar" containerID="cri-o://bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" gracePeriod=600 Nov 03 07:54:40.269738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269634 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="config-reloader" containerID="cri-o://232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" gracePeriod=600 Nov 03 07:54:40.269738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269670 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" gracePeriod=600 Nov 03 07:54:40.269970 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269570 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="prometheus" containerID="cri-o://1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" gracePeriod=600 Nov 03 07:54:40.270080 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.269804 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-web" containerID="cri-o://f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" gracePeriod=600 Nov 03 07:54:40.499499 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.499473 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.556821 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556794 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.556821 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556833 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556853 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556880 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556910 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556937 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.556963 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557006 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557033 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557065 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557108 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557151 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557183 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557222 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557250 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557241 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.557347 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557324 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557357 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557396 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557445 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557476 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") pod \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\" (UID: \"902ad59c-068f-42e8-aa8d-d1658a31e3bf\") " Nov 03 07:54:40.557996 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557877 5951 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.558053 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.557991 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.558184 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.558053 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.559353 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.559321 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.560280 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.560258 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.560698 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.560304 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:54:40.560791 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.560693 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.560791 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.560761 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:54:40.561351 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.561307 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.561712 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.561632 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.561805 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.561778 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "kube-api-access-r5wjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:54:40.562018 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.561987 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.562267 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.562249 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.562839 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.562803 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.563289 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.563265 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.563388 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.563370 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:54:40.563508 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.563485 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:54:40.564268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.564246 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.564350 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.564320 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.573498 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.573478 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config" pod "902ad59c-068f-42e8-aa8d-d1658a31e3bf" (UID: "902ad59c-068f-42e8-aa8d-d1658a31e3bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:54:40.658734 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658694 5951 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config-out\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658734 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658730 5951 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658734 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658741 5951 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-metrics-client-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658734 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658750 5951 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-grpc-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658761 5951 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658790 5951 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-web-config\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658800 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5wjs\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-kube-api-access-r5wjs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658808 5951 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-db\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658817 5951 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658826 5951 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658835 5951 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658842 5951 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658850 5951 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-config\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658858 5951 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-metrics-client-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658866 5951 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658875 5951 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658883 5951 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/902ad59c-068f-42e8-aa8d-d1658a31e3bf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658891 5951 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/902ad59c-068f-42e8-aa8d-d1658a31e3bf-tls-assets\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.658954 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.658899 5951 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/902ad59c-068f-42e8-aa8d-d1658a31e3bf-secret-kube-rbac-proxy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:54:40.852658 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852569 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" exitCode=0 Nov 03 07:54:40.852658 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852596 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" exitCode=0 Nov 03 07:54:40.852658 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852602 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" exitCode=0 Nov 03 07:54:40.852658 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852608 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" exitCode=0 Nov 03 07:54:40.852658 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852613 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" exitCode=0 Nov 03 07:54:40.852658 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852621 5951 generic.go:356] "Generic (PLEG): container finished" podID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" exitCode=0 Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852677 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852667 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852823 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852854 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852886 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852892 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852912 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852944 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} Nov 03 07:54:40.853009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.852984 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"902ad59c-068f-42e8-aa8d-d1658a31e3bf","Type":"ContainerDied","Data":"4dfc6e47475f4b35f72188d9e1459465b9e86262c700fd557422324c5b460bdc"} Nov 03 07:54:40.861759 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.861743 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.868273 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.868245 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.869178 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.869159 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:40.874258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.874236 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:40.877501 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.877482 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.883872 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.883851 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.890821 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.890789 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.895554 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.895505 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:40.896316 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896273 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="prometheus" Nov 03 07:54:40.896316 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896294 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="prometheus" Nov 03 07:54:40.896316 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896311 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy" Nov 03 07:54:40.896316 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896318 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896332 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="init-config-reloader" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896339 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="init-config-reloader" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896350 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-thanos" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896358 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-thanos" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896368 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="config-reloader" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896374 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="config-reloader" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896384 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="thanos-sidecar" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896392 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="thanos-sidecar" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896402 5951 cpu_manager.go:414] "containerMap: removing container" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-web" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896407 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-web" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896453 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896460 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="init-config-reloader" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896466 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-thanos" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896471 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="config-reloader" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896476 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="thanos-sidecar" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896480 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="kube-rbac-proxy-web" Nov 03 07:54:40.896574 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.896489 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" containerName="prometheus" Nov 03 07:54:40.898054 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.898035 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.900956 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.900937 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.904561 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904418 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 07:54:40.904561 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904481 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 07:54:40.904561 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904489 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 07:54:40.904561 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904524 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 07:54:40.904561 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904538 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 07:54:40.904893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904565 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 07:54:40.904893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904675 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 07:54:40.904893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904709 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 07:54:40.904893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904727 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 07:54:40.904893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904753 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 07:54:40.904893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904861 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bxcxf\"" Nov 03 07:54:40.905159 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904944 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8f9uviit3bo6o\"" Nov 03 07:54:40.905159 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904982 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 07:54:40.905159 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.904998 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 07:54:40.905263 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.905180 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 07:54:40.906261 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.906242 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 07:54:40.906491 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.906473 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 07:54:40.908108 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.908087 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.908397 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.908377 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.908483 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.908439 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} err="failed to get container status \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" Nov 03 07:54:40.908483 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.908476 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.908805 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.908786 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.908879 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.908811 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} err="failed to get container status \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" Nov 03 07:54:40.908879 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.908826 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.909116 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.909098 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.909165 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909121 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} err="failed to get container status \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" Nov 03 07:54:40.909165 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909136 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.909397 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.909381 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.909453 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909402 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} err="failed to get container status \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" Nov 03 07:54:40.909453 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909418 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.909686 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.909671 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.909751 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909691 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} err="failed to get container status \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" Nov 03 07:54:40.909751 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909704 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.909974 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.909960 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.910022 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909986 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} err="failed to get container status \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" Nov 03 07:54:40.910022 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.909998 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.910237 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:54:40.910217 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.910272 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910243 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} err="failed to get container status \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" Nov 03 07:54:40.910272 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910263 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.910543 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910512 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} err="failed to get container status \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" Nov 03 07:54:40.910543 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910535 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.910786 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910767 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} err="failed to get container status \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" Nov 03 07:54:40.910847 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910786 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.910996 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910981 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} err="failed to get container status \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" Nov 03 07:54:40.911046 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.910995 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.911186 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911167 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} err="failed to get container status \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" Nov 03 07:54:40.911255 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911187 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.911399 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911377 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} err="failed to get container status \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" Nov 03 07:54:40.911461 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911399 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.911594 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911577 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} err="failed to get container status \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" Nov 03 07:54:40.911680 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911596 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.911850 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911830 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} err="failed to get container status \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" Nov 03 07:54:40.911923 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.911851 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.912070 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912053 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} err="failed to get container status \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" Nov 03 07:54:40.912128 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912070 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.912244 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912223 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} err="failed to get container status \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" Nov 03 07:54:40.912244 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912241 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.912482 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912464 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} err="failed to get container status \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" Nov 03 07:54:40.912523 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912483 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.912732 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912716 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} err="failed to get container status \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" Nov 03 07:54:40.912732 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912732 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.912967 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912947 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} err="failed to get container status \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" Nov 03 07:54:40.912967 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.912965 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.913176 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913159 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} err="failed to get container status \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" Nov 03 07:54:40.913219 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913176 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.913406 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913385 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} err="failed to get container status \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" Nov 03 07:54:40.913479 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913408 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.913617 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913601 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} err="failed to get container status \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" Nov 03 07:54:40.913617 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913616 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.913853 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913835 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} err="failed to get container status \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" Nov 03 07:54:40.913919 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.913854 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.914147 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914122 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} err="failed to get container status \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" Nov 03 07:54:40.914243 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914148 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.914496 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914470 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} err="failed to get container status \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" Nov 03 07:54:40.914496 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914496 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.914739 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914720 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} err="failed to get container status \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" Nov 03 07:54:40.914794 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914739 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.914953 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914933 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} err="failed to get container status \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" Nov 03 07:54:40.915006 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.914954 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.915198 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915182 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} err="failed to get container status \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" Nov 03 07:54:40.915198 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915196 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.915424 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915404 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} err="failed to get container status \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" Nov 03 07:54:40.915495 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915425 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.915673 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915637 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} err="failed to get container status \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" Nov 03 07:54:40.915738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915674 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.915898 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915879 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} err="failed to get container status \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" Nov 03 07:54:40.915940 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.915901 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.916119 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916103 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} err="failed to get container status \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" Nov 03 07:54:40.916185 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916120 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.916382 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916361 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} err="failed to get container status \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" Nov 03 07:54:40.916430 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916383 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.916610 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916593 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} err="failed to get container status \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" Nov 03 07:54:40.916711 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916612 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.916909 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916869 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} err="failed to get container status \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" Nov 03 07:54:40.916975 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.916894 5951 scope.go:122] "RemoveContainer" containerID="457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5" Nov 03 07:54:40.917286 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917257 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5"} err="failed to get container status \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": rpc error: code = NotFound desc = could not find container \"457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5\": container with ID starting with 457c97f278c3adea1cd549c384b4de78eea0212b882a1194d704c452459fffc5 not found: ID does not exist" Nov 03 07:54:40.917286 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917283 5951 scope.go:122] "RemoveContainer" containerID="e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541" Nov 03 07:54:40.917730 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917583 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541"} err="failed to get container status \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": rpc error: code = NotFound desc = could not find container \"e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541\": container with ID starting with e775501456c773b8d54f9dcfc0f18019303c3e51868378d57c599a5c2b389541 not found: ID does not exist" Nov 03 07:54:40.917730 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917607 5951 scope.go:122] "RemoveContainer" containerID="f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65" Nov 03 07:54:40.918048 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917959 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65"} err="failed to get container status \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": rpc error: code = NotFound desc = could not find container \"f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65\": container with ID starting with f7cac3c8f4c414f515f3c5aa86f59a4a86d7bd61227dc477906644c16bf02a65 not found: ID does not exist" Nov 03 07:54:40.918048 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917980 5951 scope.go:122] "RemoveContainer" containerID="bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7" Nov 03 07:54:40.918048 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.917987 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:40.918871 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.918760 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7"} err="failed to get container status \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": rpc error: code = NotFound desc = could not find container \"bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7\": container with ID starting with bb759e1e1766f87f3429a01ba3ca9a93e381402916d75b0e1e23590ecf9d5aa7 not found: ID does not exist" Nov 03 07:54:40.918871 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.918796 5951 scope.go:122] "RemoveContainer" containerID="232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a" Nov 03 07:54:40.919081 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.919054 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a"} err="failed to get container status \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": rpc error: code = NotFound desc = could not find container \"232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a\": container with ID starting with 232211de226f074a8237bec58adf36062e28cce9c281b8a884b9da809a00bc4a not found: ID does not exist" Nov 03 07:54:40.919144 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.919082 5951 scope.go:122] "RemoveContainer" containerID="1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7" Nov 03 07:54:40.919866 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.919839 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7"} err="failed to get container status \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": rpc error: code = NotFound desc = could not find container \"1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7\": container with ID starting with 1caad9f9850b9e54747c65f74b700c15ecd2737d1acfef646c91ad3431810eb7 not found: ID does not exist" Nov 03 07:54:40.919866 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.919865 5951 scope.go:122] "RemoveContainer" containerID="da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f" Nov 03 07:54:40.920177 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.920151 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f"} err="failed to get container status \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": rpc error: code = NotFound desc = could not find container \"da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f\": container with ID starting with da5bbf5f5231bd158be00f0cb21ccab024fdae816f6f2780d09ce043e403054f not found: ID does not exist" Nov 03 07:54:40.960748 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.960721 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.960885 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.960753 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.960885 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.960770 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtjw\" (UniqueName: \"kubernetes.io/projected/f7811fd3-c92a-4c78-86bd-c4c06334941c-kube-api-access-pwtjw\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.960885 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.960789 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7811fd3-c92a-4c78-86bd-c4c06334941c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961018 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.960969 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961018 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.960998 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961110 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961018 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961110 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961040 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961175 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961113 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7811fd3-c92a-4c78-86bd-c4c06334941c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961175 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961136 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961175 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961161 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961284 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961193 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961284 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961229 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-config\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961284 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961258 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961284 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961279 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961409 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961300 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961409 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961348 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961409 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961366 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961409 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961382 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:40.961542 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:40.961409 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062426 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062398 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062426 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062427 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062617 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062444 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtjw\" (UniqueName: \"kubernetes.io/projected/f7811fd3-c92a-4c78-86bd-c4c06334941c-kube-api-access-pwtjw\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062617 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062461 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7811fd3-c92a-4c78-86bd-c4c06334941c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062617 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062591 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062623 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062638 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062686 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062714 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7811fd3-c92a-4c78-86bd-c4c06334941c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062741 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.062783 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062776 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062803 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062825 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-config\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062855 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062855 5951 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062880 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062904 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062936 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062961 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.062989 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.063020 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063577 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.063266 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063577 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.063303 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.063702 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.063611 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.064799 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.064035 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.064799 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.064479 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.065605 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.065578 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7811fd3-c92a-4c78-86bd-c4c06334941c-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.065707 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.063309 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.065770 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.065722 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7811fd3-c92a-4c78-86bd-c4c06334941c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.066338 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.066313 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.066436 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.066378 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.066527 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.066502 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.067083 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.067014 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.067180 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.067149 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.067606 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.067578 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7811fd3-c92a-4c78-86bd-c4c06334941c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.067785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.067764 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.068458 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.068436 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-config\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.068527 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.068460 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.068764 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.068741 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.068864 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.068848 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7811fd3-c92a-4c78-86bd-c4c06334941c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.069564 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.069549 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtjw\" (UniqueName: \"kubernetes.io/projected/f7811fd3-c92a-4c78-86bd-c4c06334941c-kube-api-access-pwtjw\") pod \"prometheus-k8s-0\" (UID: \"f7811fd3-c92a-4c78-86bd-c4c06334941c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.228863 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.228802 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:41.503341 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.503260 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 07:54:41.507327 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:54:41.507297 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7811fd3_c92a_4c78_86bd_c4c06334941c.slice/crio-84378d9b0600fbea9d822f1c55029da1bca0cdeec273991e24f2f561691ffe37 WatchSource:0}: Error finding container 84378d9b0600fbea9d822f1c55029da1bca0cdeec273991e24f2f561691ffe37: Status 404 returned error can't find the container with id 84378d9b0600fbea9d822f1c55029da1bca0cdeec273991e24f2f561691ffe37 Nov 03 07:54:41.858289 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.858257 5951 generic.go:356] "Generic (PLEG): container finished" podID="f7811fd3-c92a-4c78-86bd-c4c06334941c" containerID="bbfb9371ad6806f76308c6d9df97b730c3ef23af809da2cfbb27e211ad0bae43" exitCode=0 Nov 03 07:54:41.858436 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.858346 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerDied","Data":"bbfb9371ad6806f76308c6d9df97b730c3ef23af809da2cfbb27e211ad0bae43"} Nov 03 07:54:41.858511 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:41.858436 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"84378d9b0600fbea9d822f1c55029da1bca0cdeec273991e24f2f561691ffe37"} Nov 03 07:54:42.733632 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.733601 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902ad59c-068f-42e8-aa8d-d1658a31e3bf" path="/var/lib/kubelet/pods/902ad59c-068f-42e8-aa8d-d1658a31e3bf/volumes" Nov 03 07:54:42.866762 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.866731 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"2515105a8c5a34fec999eb6ec45e925cf542f7ca4888e4f3c1402c4fd115f803"} Nov 03 07:54:42.866884 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.866792 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"d386c810018b41a4f7862fa1fd8d0c90807224f9a1c748b4c7454114e0f49518"} Nov 03 07:54:42.866884 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.866817 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"64dec5e0b2e456d82ceaac56ef5ab7cad1120e125564b14b3e5746bdb5c742c5"} Nov 03 07:54:42.866884 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.866844 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"11ff9d73e588c0aa3bf411ccd35bd237133f67ab7745c6343ec084425a69becc"} Nov 03 07:54:42.866884 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.866867 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"0e58c8eb1ec51ff2fa3781bf2e65e97a49397aa470bf38fd274cc0b39e568cd4"} Nov 03 07:54:42.867042 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.866888 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7811fd3-c92a-4c78-86bd-c4c06334941c","Type":"ContainerStarted","Data":"fcad01ed070368afdaea0a0d896a86b65cea77cb8469819074e0f27225081fff"} Nov 03 07:54:42.891967 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:42.891911 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.874195307 podStartE2EDuration="2.891894267s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.69896ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:42.890665706 +0000 UTC m=+22.272116617" watchObservedRunningTime="2026-11-03 07:54:42.891894267 +0000 UTC m=+22.273345185" Nov 03 07:54:46.229312 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:46.229268 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:54:57.850611 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:54:57.850584 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-brfqb" Nov 03 07:55:01.953735 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:55:01.953701 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:55:08.471992 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:55:08.471951 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:55:08.487960 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:55:08.487935 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:55:08.977380 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:55:08.977350 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 07:55:31.953198 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:55:31.953163 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:56:01.953014 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:56:01.952932 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:56:31.952794 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:56:31.952757 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:57:01.953533 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:57:01.953488 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:57:15.093204 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.093172 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj"] Nov 03 07:57:15.096787 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.096768 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.099524 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.099498 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 07:57:15.099621 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.099572 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v9mbj\"" Nov 03 07:57:15.099621 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.099574 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 07:57:15.105974 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.105954 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj"] Nov 03 07:57:15.190785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.190747 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk"] Nov 03 07:57:15.197363 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.197336 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.201633 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.201590 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8hf5l\"" Nov 03 07:57:15.202454 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.202429 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk"] Nov 03 07:57:15.271051 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.271014 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.271051 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.271051 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.271249 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.271161 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g94v\" (UniqueName: \"kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.291602 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.291573 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn"] Nov 03 07:57:15.295170 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.295150 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.297903 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.297888 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-rxvtn\"" Nov 03 07:57:15.303502 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.303480 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn"] Nov 03 07:57:15.371735 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.371616 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nx8\" (UniqueName: \"kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.371735 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.371697 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g94v\" (UniqueName: \"kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.371984 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.371798 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.371984 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.371845 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.371984 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.371883 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.371984 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.371913 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.372206 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.372189 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.372258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.372241 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.379144 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.379108 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g94v\" (UniqueName: \"kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.392407 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.392385 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7"] Nov 03 07:57:15.396102 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.396082 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.403706 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.403684 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7"] Nov 03 07:57:15.407739 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.407722 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-n6czx\"" Nov 03 07:57:15.413797 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.413781 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473230 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nx8\" (UniqueName: \"kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473277 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473351 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473450 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzdd\" (UniqueName: \"kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473490 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473513 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473771 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.473913 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.473841 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.493444 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.493423 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nx8\" (UniqueName: \"kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.514258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.514238 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:15.574117 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574090 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.574237 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574130 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzdd\" (UniqueName: \"kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.574237 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574157 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.574349 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574238 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.574349 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574274 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.574349 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574299 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlh9p\" (UniqueName: \"kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.574599 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574581 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.574681 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.574610 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.580978 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.580955 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzdd\" (UniqueName: \"kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.611278 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.611259 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:15.675695 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.675660 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.675818 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.675710 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlh9p\" (UniqueName: \"kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.675818 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.675760 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.676038 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.676015 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.676095 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.676030 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.680979 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.680939 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj"] Nov 03 07:57:15.685078 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.685060 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlh9p\" (UniqueName: \"kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.685867 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:15.685843 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc214afd3_6566_4bb6_b67d_36aa494c2b6a.slice/crio-87ddaf5521a600c7bb506aa36cddbfa5f23489774ae79fc34e5980e0564da041 WatchSource:0}: Error finding container 87ddaf5521a600c7bb506aa36cddbfa5f23489774ae79fc34e5980e0564da041: Status 404 returned error can't find the container with id 87ddaf5521a600c7bb506aa36cddbfa5f23489774ae79fc34e5980e0564da041 Nov 03 07:57:15.687862 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.687847 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:57:15.712188 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.712166 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:15.787889 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.783889 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk"] Nov 03 07:57:15.791727 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:15.789724 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f46347_0226_409d_bcf4_fcf4d023763c.slice/crio-8c14029775603adb9b050a6690f5cebfa75cace9c8a4348ba7cfd1a1021ad3ba WatchSource:0}: Error finding container 8c14029775603adb9b050a6690f5cebfa75cace9c8a4348ba7cfd1a1021ad3ba: Status 404 returned error can't find the container with id 8c14029775603adb9b050a6690f5cebfa75cace9c8a4348ba7cfd1a1021ad3ba Nov 03 07:57:15.878984 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.878963 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn"] Nov 03 07:57:15.881190 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:15.881166 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452c96b0_e181_4728_a0ba_49f15660d9a0.slice/crio-1c267066fdc9df9f2d17294a04cbe1509ded91b02ec8335fe28f7948eac2db1c WatchSource:0}: Error finding container 1c267066fdc9df9f2d17294a04cbe1509ded91b02ec8335fe28f7948eac2db1c: Status 404 returned error can't find the container with id 1c267066fdc9df9f2d17294a04cbe1509ded91b02ec8335fe28f7948eac2db1c Nov 03 07:57:15.979072 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:15.978991 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7"] Nov 03 07:57:15.982834 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:15.982800 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa7fa9a_2d1b_4390_a2c7_ebefd68da915.slice/crio-9ad6096a09e6f9df97d0f0af4c00effa722cdfa91ecd1c1a2acfba4ea92e964f WatchSource:0}: Error finding container 9ad6096a09e6f9df97d0f0af4c00effa722cdfa91ecd1c1a2acfba4ea92e964f: Status 404 returned error can't find the container with id 9ad6096a09e6f9df97d0f0af4c00effa722cdfa91ecd1c1a2acfba4ea92e964f Nov 03 07:57:16.373351 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.373316 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" event={"ID":"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915","Type":"ContainerStarted","Data":"9ad6096a09e6f9df97d0f0af4c00effa722cdfa91ecd1c1a2acfba4ea92e964f"} Nov 03 07:57:16.374511 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.374485 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerStarted","Data":"8c14029775603adb9b050a6690f5cebfa75cace9c8a4348ba7cfd1a1021ad3ba"} Nov 03 07:57:16.375946 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.375923 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" event={"ID":"c214afd3-6566-4bb6-b67d-36aa494c2b6a","Type":"ContainerStarted","Data":"87ddaf5521a600c7bb506aa36cddbfa5f23489774ae79fc34e5980e0564da041"} Nov 03 07:57:16.377284 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.377258 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" event={"ID":"452c96b0-e181-4728-a0ba-49f15660d9a0","Type":"ContainerStarted","Data":"1c267066fdc9df9f2d17294a04cbe1509ded91b02ec8335fe28f7948eac2db1c"} Nov 03 07:57:16.868911 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.868877 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl"] Nov 03 07:57:16.872842 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.872813 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.875501 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.875470 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-p24hw\"" Nov 03 07:57:16.880814 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.880789 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl"] Nov 03 07:57:16.893516 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.892791 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.893516 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.892852 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.893516 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.892874 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctdh\" (UniqueName: \"kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.993497 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.993464 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.993690 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.993509 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctdh\" (UniqueName: \"kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.993690 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.993556 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.993956 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.993902 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:16.993956 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:16.993936 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:17.001956 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.001930 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctdh\" (UniqueName: \"kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:17.195339 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.195255 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:17.486624 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.486498 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl"] Nov 03 07:57:17.491592 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:17.491563 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c3c847_1f11_4c84_b8b2_7a1f9047ed19.slice/crio-f9209155a9df5326638a37fc08249c14230376fe39a1e56c938710d2fa96b350 WatchSource:0}: Error finding container f9209155a9df5326638a37fc08249c14230376fe39a1e56c938710d2fa96b350: Status 404 returned error can't find the container with id f9209155a9df5326638a37fc08249c14230376fe39a1e56c938710d2fa96b350 Nov 03 07:57:17.578718 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.578384 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm"] Nov 03 07:57:17.583718 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.583688 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.589344 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.589286 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm"] Nov 03 07:57:17.593096 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.589972 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7crpn\"" Nov 03 07:57:17.704248 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.704218 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfgw\" (UniqueName: \"kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.704421 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.704259 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.704421 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.704287 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.805382 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.805339 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfgw\" (UniqueName: \"kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.805562 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.805396 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.805562 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.805426 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.806123 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.806100 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.806229 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.806103 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.813527 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.813497 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfgw\" (UniqueName: \"kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:17.910788 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:17.910751 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:18.195797 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:18.195697 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm"] Nov 03 07:57:18.199692 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:18.199659 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396f7685_2ddb_43ce_95b2_30f3b1b8aab2.slice/crio-dfc87b0b4e03575bed51efdd1bc85b69e320f462113412b706e19d629b0811bd WatchSource:0}: Error finding container dfc87b0b4e03575bed51efdd1bc85b69e320f462113412b706e19d629b0811bd: Status 404 returned error can't find the container with id dfc87b0b4e03575bed51efdd1bc85b69e320f462113412b706e19d629b0811bd Nov 03 07:57:18.389090 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:18.388996 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" event={"ID":"396f7685-2ddb-43ce-95b2-30f3b1b8aab2","Type":"ContainerStarted","Data":"dfc87b0b4e03575bed51efdd1bc85b69e320f462113412b706e19d629b0811bd"} Nov 03 07:57:18.391767 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:18.391692 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" event={"ID":"73c3c847-1f11-4c84-b8b2-7a1f9047ed19","Type":"ContainerStarted","Data":"f9209155a9df5326638a37fc08249c14230376fe39a1e56c938710d2fa96b350"} Nov 03 07:57:20.406545 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.405896 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w"] Nov 03 07:57:20.411984 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.410983 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.435799 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.434527 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w"] Nov 03 07:57:20.435799 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.434760 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8mwhh\"" Nov 03 07:57:20.534307 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.534205 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.534307 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.534309 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.534565 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.534342 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dpk\" (UniqueName: \"kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.635412 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.635366 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.635614 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.635425 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dpk\" (UniqueName: \"kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.635614 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.635473 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.635901 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.635882 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.636092 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.636074 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.649669 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.649617 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dpk\" (UniqueName: \"kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.743881 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.742217 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8mwhh\"" Nov 03 07:57:20.751300 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.750987 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:20.874104 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.874068 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c"] Nov 03 07:57:20.880041 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.879242 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:20.883498 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.883181 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 07:57:20.883498 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.883287 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 07:57:20.883498 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.883388 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 07:57:20.883498 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.883474 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zg2vp\"" Nov 03 07:57:20.898845 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.898818 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c"] Nov 03 07:57:20.942378 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.942343 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk"] Nov 03 07:57:20.946288 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.946262 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:20.960181 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.960135 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk"] Nov 03 07:57:20.982789 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:20.982759 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-9654f\"" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039224 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039290 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039373 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039400 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9mw\" (UniqueName: \"kubernetes.io/projected/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-kube-api-access-cz9mw\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039429 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039452 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039476 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039502 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.039685 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.039519 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.046460 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.046428 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w"] Nov 03 07:57:21.056766 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:21.056733 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a30be0e_db12_4a4d_b95b_fac92e55ea8d.slice/crio-0e6db9f63f8d5dd6d2769375455f40ba5451970ab035ad962f444c5de0293d7e WatchSource:0}: Error finding container 0e6db9f63f8d5dd6d2769375455f40ba5451970ab035ad962f444c5de0293d7e: Status 404 returned error can't find the container with id 0e6db9f63f8d5dd6d2769375455f40ba5451970ab035ad962f444c5de0293d7e Nov 03 07:57:21.140099 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140036 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140105 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140139 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140166 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947a010-5e5c-462b-9739-6fd678b0c6ab-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140206 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9hq\" (UniqueName: \"kubernetes.io/projected/7947a010-5e5c-462b-9739-6fd678b0c6ab-kube-api-access-rs9hq\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140233 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140270 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140304 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140343 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140371 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9mw\" (UniqueName: \"kubernetes.io/projected/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-kube-api-access-cz9mw\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140398 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140434 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140460 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140493 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.140595 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140555 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.141034 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140595 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.141034 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140622 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.141034 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.140663 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.141200 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.141042 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.141815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.141267 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.141815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.141316 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.141815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.141537 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.141815 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.141588 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.144428 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.144399 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.144908 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.144890 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.150675 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.150631 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.154035 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.152555 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9mw\" (UniqueName: \"kubernetes.io/projected/dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1-kube-api-access-cz9mw\") pod \"maas-default-gateway-openshift-default-76f4876667-sn42c\" (UID: \"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.203821 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.203792 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:21.241656 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.241607 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.241805 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.241689 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.241805 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.241722 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.241805 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.241791 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.242008 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.241978 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947a010-5e5c-462b-9739-6fd678b0c6ab-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.242060 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.242043 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9hq\" (UniqueName: \"kubernetes.io/projected/7947a010-5e5c-462b-9739-6fd678b0c6ab-kube-api-access-rs9hq\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.242100 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.242072 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.242134 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.242121 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.242167 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.242155 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.242588 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.242564 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.243411 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.243111 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.243411 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.243337 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.243411 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.243343 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7947a010-5e5c-462b-9739-6fd678b0c6ab-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.244109 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.243848 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.250742 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.245623 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.250742 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.247178 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.252279 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.252205 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9hq\" (UniqueName: \"kubernetes.io/projected/7947a010-5e5c-462b-9739-6fd678b0c6ab-kube-api-access-rs9hq\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.254085 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.254061 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7947a010-5e5c-462b-9739-6fd678b0c6ab-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk\" (UID: \"7947a010-5e5c-462b-9739-6fd678b0c6ab\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.269131 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.268296 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:21.476974 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.476813 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" event={"ID":"2a30be0e-db12-4a4d-b95b-fac92e55ea8d","Type":"ContainerStarted","Data":"0e6db9f63f8d5dd6d2769375455f40ba5451970ab035ad962f444c5de0293d7e"} Nov 03 07:57:21.499174 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.499059 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c"] Nov 03 07:57:21.503111 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:21.503068 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1ad89_4520_43b0_b7c6_b7dcb10ee2b1.slice/crio-db5ff00d205c257a56392fc66a69d619705d15a9e1ac46e4218e67366bc02bc5 WatchSource:0}: Error finding container db5ff00d205c257a56392fc66a69d619705d15a9e1ac46e4218e67366bc02bc5: Status 404 returned error can't find the container with id db5ff00d205c257a56392fc66a69d619705d15a9e1ac46e4218e67366bc02bc5 Nov 03 07:57:21.588299 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:21.588221 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk"] Nov 03 07:57:22.492885 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.492842 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" event={"ID":"7947a010-5e5c-462b-9739-6fd678b0c6ab","Type":"ContainerStarted","Data":"09fd5683ee3022f3418cb9af4b50bbeb6fd1da9dd125c71862fff82b10417e8d"} Nov 03 07:57:22.498005 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.497972 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" event={"ID":"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1","Type":"ContainerStarted","Data":"db5ff00d205c257a56392fc66a69d619705d15a9e1ac46e4218e67366bc02bc5"} Nov 03 07:57:22.555587 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.555541 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh"] Nov 03 07:57:22.559746 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.559721 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.568130 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.568101 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh"] Nov 03 07:57:22.656197 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.656153 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.656466 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.656212 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ksq\" (UniqueName: \"kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.656466 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.656282 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.759164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.757685 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.759164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.757822 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.759164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.757857 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ksq\" (UniqueName: \"kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.759164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.758685 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.759164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.759044 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.768003 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.767950 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ksq\" (UniqueName: \"kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:22.886421 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:22.886389 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:57:23.225842 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:23.225692 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh"] Nov 03 07:57:23.537388 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:23.537348 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" event={"ID":"fe19772c-5c94-4231-a786-a2545ef6362b","Type":"ContainerStarted","Data":"ed9adaaf3fd0fc65558965b9cf9cacc8efc6205a5284540966333135d6c4bbf2"} Nov 03 07:57:31.029891 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.029857 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:31.030267 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.029932 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:31.030267 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.029966 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:31.586080 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.585182 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:31.586080 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.585279 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:31.586080 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.585325 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:31.615838 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.615786 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" event={"ID":"7947a010-5e5c-462b-9739-6fd678b0c6ab","Type":"ContainerStarted","Data":"3af53bb8680c5d273a0e0d4817ca5fde7096b9caed53fb8f16be4b6050e057ed"} Nov 03 07:57:31.637007 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:31.636929 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" podStartSLOduration=2.217404121 podStartE2EDuration="11.636908342s" totalImagesPullingTime="9.419504221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:31.634874554 +0000 UTC m=+191.016325471" watchObservedRunningTime="2026-11-03 07:57:31.636908342 +0000 UTC m=+191.018359274" Nov 03 07:57:32.000341 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:57:32.000296 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:57:32.269539 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.269435 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:32.274819 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.274791 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:32.623175 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.623136 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" event={"ID":"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1","Type":"ContainerStarted","Data":"51df9c11cb64730a622f35a7d66eb442e4b0108b7aeceb406972868fa64f15b1"} Nov 03 07:57:32.625277 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.625249 5951 generic.go:356] "Generic (PLEG): container finished" podID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerID="f1ecd6ca904f07a7503ed1709ea2a9be9be11775876b97d640221e052e656e81" exitCode=0 Nov 03 07:57:32.625467 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.625345 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" event={"ID":"452c96b0-e181-4728-a0ba-49f15660d9a0","Type":"ContainerDied","Data":"f1ecd6ca904f07a7503ed1709ea2a9be9be11775876b97d640221e052e656e81"} Nov 03 07:57:32.628005 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.627976 5951 generic.go:356] "Generic (PLEG): container finished" podID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerID="77abbbb74ce700dd1654ca3a8412369dc429bfdf61d2d01391d65905b87df718" exitCode=0 Nov 03 07:57:32.628109 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.628068 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" event={"ID":"396f7685-2ddb-43ce-95b2-30f3b1b8aab2","Type":"ContainerDied","Data":"77abbbb74ce700dd1654ca3a8412369dc429bfdf61d2d01391d65905b87df718"} Nov 03 07:57:32.630872 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.630846 5951 generic.go:356] "Generic (PLEG): container finished" podID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerID="31ca67fff20ccf504647afda3aec1d45f1e0dd6ea4f7a0b4b0c9e0aeabaffbdb" exitCode=0 Nov 03 07:57:32.630970 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.630912 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" event={"ID":"2a30be0e-db12-4a4d-b95b-fac92e55ea8d","Type":"ContainerDied","Data":"31ca67fff20ccf504647afda3aec1d45f1e0dd6ea4f7a0b4b0c9e0aeabaffbdb"} Nov 03 07:57:32.631498 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.631319 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:32.632269 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.632253 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk" Nov 03 07:57:32.640308 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:32.640264 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podStartSLOduration=2.561699064 podStartE2EDuration="12.640244106s" totalImagesPullingTime="10.078545042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:32.639072644 +0000 UTC m=+192.020523561" watchObservedRunningTime="2026-11-03 07:57:32.640244106 +0000 UTC m=+192.021695017" Nov 03 07:57:33.204247 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.204215 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:33.209771 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.209747 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:33.637230 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.637140 5951 generic.go:356] "Generic (PLEG): container finished" podID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerID="c083d1ba7625d8938470493e15fd517398176f5e9cb53c4ccd7e291a9edaa879" exitCode=0 Nov 03 07:57:33.637706 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.637241 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" event={"ID":"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915","Type":"ContainerDied","Data":"c083d1ba7625d8938470493e15fd517398176f5e9cb53c4ccd7e291a9edaa879"} Nov 03 07:57:33.639771 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.639746 5951 generic.go:356] "Generic (PLEG): container finished" podID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerID="bbccab2545eddff672e08d3ee881338991e05a63aaf0139410e1dc9937e01e55" exitCode=0 Nov 03 07:57:33.639875 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.639836 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" event={"ID":"73c3c847-1f11-4c84-b8b2-7a1f9047ed19","Type":"ContainerDied","Data":"bbccab2545eddff672e08d3ee881338991e05a63aaf0139410e1dc9937e01e55"} Nov 03 07:57:33.641528 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.641508 5951 generic.go:356] "Generic (PLEG): container finished" podID="fe19772c-5c94-4231-a786-a2545ef6362b" containerID="3e88cc71fc1819002a518c181d07bc28c0c99983d48f92d75bf5323773168f64" exitCode=0 Nov 03 07:57:33.641622 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.641538 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" event={"ID":"fe19772c-5c94-4231-a786-a2545ef6362b","Type":"ContainerDied","Data":"3e88cc71fc1819002a518c181d07bc28c0c99983d48f92d75bf5323773168f64"} Nov 03 07:57:33.643105 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.643083 5951 generic.go:356] "Generic (PLEG): container finished" podID="01f46347-0226-409d-bcf4-fcf4d023763c" containerID="193d48714a50de05b89f082560b1233be7f4c913be14ce3d3ebf7a959207b8e8" exitCode=0 Nov 03 07:57:33.643208 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.643144 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerDied","Data":"193d48714a50de05b89f082560b1233be7f4c913be14ce3d3ebf7a959207b8e8"} Nov 03 07:57:33.644778 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.644704 5951 generic.go:356] "Generic (PLEG): container finished" podID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerID="41d7d85eb562c53e19cf8d38863b72c5b322980e3bda3f0372aba09f8f5e066f" exitCode=0 Nov 03 07:57:33.644835 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.644802 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" event={"ID":"c214afd3-6566-4bb6-b67d-36aa494c2b6a","Type":"ContainerDied","Data":"41d7d85eb562c53e19cf8d38863b72c5b322980e3bda3f0372aba09f8f5e066f"} Nov 03 07:57:33.645269 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.645116 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:33.646087 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:33.646067 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 07:57:34.653249 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.653177 5951 generic.go:356] "Generic (PLEG): container finished" podID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerID="f537cb0651380259c42942e385094b74f6ab9b3ef45bf56950b84a719365b62d" exitCode=0 Nov 03 07:57:34.653678 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.653252 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" event={"ID":"73c3c847-1f11-4c84-b8b2-7a1f9047ed19","Type":"ContainerDied","Data":"f537cb0651380259c42942e385094b74f6ab9b3ef45bf56950b84a719365b62d"} Nov 03 07:57:34.656817 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.656792 5951 generic.go:356] "Generic (PLEG): container finished" podID="fe19772c-5c94-4231-a786-a2545ef6362b" containerID="b5cf1f852b79cafc27888f6f8da020e0a73e183af971e980ef0b6ef9f44c5f33" exitCode=0 Nov 03 07:57:34.656958 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.656892 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" event={"ID":"fe19772c-5c94-4231-a786-a2545ef6362b","Type":"ContainerDied","Data":"b5cf1f852b79cafc27888f6f8da020e0a73e183af971e980ef0b6ef9f44c5f33"} Nov 03 07:57:34.659286 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.659262 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerStarted","Data":"c0c2d49e5af6a965f5fa8603c1ff02d9a9bfb1f063882590d60be147785fafcb"} Nov 03 07:57:34.662057 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.662019 5951 generic.go:356] "Generic (PLEG): container finished" podID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerID="3ca4738f8cd5ecbcb44231728c8543a843063d958e7f1853d459f12a44c6dde7" exitCode=0 Nov 03 07:57:34.662733 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:34.662710 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" event={"ID":"452c96b0-e181-4728-a0ba-49f15660d9a0","Type":"ContainerDied","Data":"3ca4738f8cd5ecbcb44231728c8543a843063d958e7f1853d459f12a44c6dde7"} Nov 03 07:57:35.674216 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.674132 5951 generic.go:356] "Generic (PLEG): container finished" podID="01f46347-0226-409d-bcf4-fcf4d023763c" containerID="c0c2d49e5af6a965f5fa8603c1ff02d9a9bfb1f063882590d60be147785fafcb" exitCode=0 Nov 03 07:57:35.674860 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.674223 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerDied","Data":"c0c2d49e5af6a965f5fa8603c1ff02d9a9bfb1f063882590d60be147785fafcb"} Nov 03 07:57:35.680052 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.680023 5951 generic.go:356] "Generic (PLEG): container finished" podID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerID="601be671b6a31be98839f7df7c8655d2d3c9c16f10615edacba2cd52313359dd" exitCode=0 Nov 03 07:57:35.680195 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.680164 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" event={"ID":"c214afd3-6566-4bb6-b67d-36aa494c2b6a","Type":"ContainerDied","Data":"601be671b6a31be98839f7df7c8655d2d3c9c16f10615edacba2cd52313359dd"} Nov 03 07:57:35.684947 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.684906 5951 generic.go:356] "Generic (PLEG): container finished" podID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerID="451a4cd7f2776c695844c8f1da543e9da4d93e63fef52f45eeb6799178341486" exitCode=0 Nov 03 07:57:35.685122 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.685099 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" event={"ID":"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915","Type":"ContainerDied","Data":"451a4cd7f2776c695844c8f1da543e9da4d93e63fef52f45eeb6799178341486"} Nov 03 07:57:35.691695 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.691589 5951 generic.go:356] "Generic (PLEG): container finished" podID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerID="efadecebfbae35b9244efc968ac9f81ff8b0fd3681bd61f5800b2526e6536347" exitCode=0 Nov 03 07:57:35.691695 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:35.691623 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" event={"ID":"2a30be0e-db12-4a4d-b95b-fac92e55ea8d","Type":"ContainerDied","Data":"efadecebfbae35b9244efc968ac9f81ff8b0fd3681bd61f5800b2526e6536347"} Nov 03 07:57:36.709698 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:36.708711 5951 generic.go:356] "Generic (PLEG): container finished" podID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerID="3c2510e6b8c3d8f4236fece9411fd719f22286fd95513d4012f264abead34fac" exitCode=0 Nov 03 07:57:36.709698 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:36.708774 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" event={"ID":"396f7685-2ddb-43ce-95b2-30f3b1b8aab2","Type":"ContainerDied","Data":"3c2510e6b8c3d8f4236fece9411fd719f22286fd95513d4012f264abead34fac"} Nov 03 07:57:50.838895 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:50.838393 5951 generic.go:356] "Generic (PLEG): container finished" podID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerID="d044217f99383b3f34ffafbe794b0a0196d661b705482b4b41050286b7656d63" exitCode=0 Nov 03 07:57:50.838895 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:50.838488 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" event={"ID":"73c3c847-1f11-4c84-b8b2-7a1f9047ed19","Type":"ContainerDied","Data":"d044217f99383b3f34ffafbe794b0a0196d661b705482b4b41050286b7656d63"} Nov 03 07:57:51.849155 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:51.848574 5951 generic.go:356] "Generic (PLEG): container finished" podID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerID="a89b55c233dc1d52e2c6d5a4c9730c84d8f4c30daa58007559adb3e46125d80e" exitCode=0 Nov 03 07:57:51.849155 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:51.848973 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" event={"ID":"452c96b0-e181-4728-a0ba-49f15660d9a0","Type":"ContainerDied","Data":"a89b55c233dc1d52e2c6d5a4c9730c84d8f4c30daa58007559adb3e46125d80e"} Nov 03 07:57:51.852263 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:51.851723 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerStarted","Data":"bcaf04fe9336e4dbbcb7a2f2e13700c974b191f306ba53d496cc1bf196b3950b"} Nov 03 07:57:51.881473 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:51.880816 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" podStartSLOduration=2.801420295 podStartE2EDuration="36.880800793s" totalImagesPullingTime="34.079380498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:51.880310927 +0000 UTC m=+211.261761842" watchObservedRunningTime="2026-11-03 07:57:51.880800793 +0000 UTC m=+211.262251689" Nov 03 07:57:52.070723 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.070447 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:52.177724 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.177694 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh\" (UniqueName: \"kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh\") pod \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " Nov 03 07:57:52.177874 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.177837 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util\") pod \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " Nov 03 07:57:52.177874 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.177860 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle\") pod \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\" (UID: \"73c3c847-1f11-4c84-b8b2-7a1f9047ed19\") " Nov 03 07:57:52.178534 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.178501 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle" pod "73c3c847-1f11-4c84-b8b2-7a1f9047ed19" (UID: "73c3c847-1f11-4c84-b8b2-7a1f9047ed19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:52.179990 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.179967 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh" pod "73c3c847-1f11-4c84-b8b2-7a1f9047ed19" (UID: "73c3c847-1f11-4c84-b8b2-7a1f9047ed19"). InnerVolumeSpecName "kube-api-access-9ctdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:52.183738 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.183712 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util" pod "73c3c847-1f11-4c84-b8b2-7a1f9047ed19" (UID: "73c3c847-1f11-4c84-b8b2-7a1f9047ed19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:52.278388 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.278364 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:52.278489 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.278393 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:52.278489 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.278408 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctdh\" (UniqueName: \"kubernetes.io/projected/73c3c847-1f11-4c84-b8b2-7a1f9047ed19-kube-api-access-9ctdh\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:52.858056 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.858018 5951 generic.go:356] "Generic (PLEG): container finished" podID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerID="56e3142f660681623d9b8b284643ac09558e465da6613fcb21dd1ce05ba5bcaf" exitCode=0 Nov 03 07:57:52.858445 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.858105 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" event={"ID":"396f7685-2ddb-43ce-95b2-30f3b1b8aab2","Type":"ContainerDied","Data":"56e3142f660681623d9b8b284643ac09558e465da6613fcb21dd1ce05ba5bcaf"} Nov 03 07:57:52.860266 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.860242 5951 generic.go:356] "Generic (PLEG): container finished" podID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerID="e40becf6171a06f154db12559386b6ee6e313547831f855291f84687fd77718a" exitCode=0 Nov 03 07:57:52.860379 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.860313 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" event={"ID":"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915","Type":"ContainerDied","Data":"e40becf6171a06f154db12559386b6ee6e313547831f855291f84687fd77718a"} Nov 03 07:57:52.862105 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.862085 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" event={"ID":"73c3c847-1f11-4c84-b8b2-7a1f9047ed19","Type":"ContainerDied","Data":"f9209155a9df5326638a37fc08249c14230376fe39a1e56c938710d2fa96b350"} Nov 03 07:57:52.862105 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.862102 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q6rtl" Nov 03 07:57:52.862282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.862122 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9209155a9df5326638a37fc08249c14230376fe39a1e56c938710d2fa96b350" Nov 03 07:57:52.864404 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.864370 5951 generic.go:356] "Generic (PLEG): container finished" podID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerID="096c1c140414c931ca807ed7fec270b5ae51c3852c8238ad6cc67434b3e59fda" exitCode=0 Nov 03 07:57:52.864404 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.864394 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" event={"ID":"2a30be0e-db12-4a4d-b95b-fac92e55ea8d","Type":"ContainerDied","Data":"096c1c140414c931ca807ed7fec270b5ae51c3852c8238ad6cc67434b3e59fda"} Nov 03 07:57:52.866413 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.866394 5951 generic.go:356] "Generic (PLEG): container finished" podID="01f46347-0226-409d-bcf4-fcf4d023763c" containerID="bcaf04fe9336e4dbbcb7a2f2e13700c974b191f306ba53d496cc1bf196b3950b" exitCode=0 Nov 03 07:57:52.866511 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.866460 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerDied","Data":"bcaf04fe9336e4dbbcb7a2f2e13700c974b191f306ba53d496cc1bf196b3950b"} Nov 03 07:57:52.868397 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.868378 5951 generic.go:356] "Generic (PLEG): container finished" podID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerID="c8d51c27dea3f3bf9b4bba6bab9acf0a314c334211f8ede6447ba5783cafa1cb" exitCode=0 Nov 03 07:57:52.868476 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:52.868457 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" event={"ID":"c214afd3-6566-4bb6-b67d-36aa494c2b6a","Type":"ContainerDied","Data":"c8d51c27dea3f3bf9b4bba6bab9acf0a314c334211f8ede6447ba5783cafa1cb"} Nov 03 07:57:53.002223 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.002163 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:53.086298 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.086261 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle\") pod \"452c96b0-e181-4728-a0ba-49f15660d9a0\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " Nov 03 07:57:53.086298 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.086309 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util\") pod \"452c96b0-e181-4728-a0ba-49f15660d9a0\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " Nov 03 07:57:53.086518 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.086335 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd\" (UniqueName: \"kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd\") pod \"452c96b0-e181-4728-a0ba-49f15660d9a0\" (UID: \"452c96b0-e181-4728-a0ba-49f15660d9a0\") " Nov 03 07:57:53.086837 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.086809 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle" pod "452c96b0-e181-4728-a0ba-49f15660d9a0" (UID: "452c96b0-e181-4728-a0ba-49f15660d9a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:53.088680 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.088627 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd" pod "452c96b0-e181-4728-a0ba-49f15660d9a0" (UID: "452c96b0-e181-4728-a0ba-49f15660d9a0"). InnerVolumeSpecName "kube-api-access-xrzdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:53.091558 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.091526 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util" pod "452c96b0-e181-4728-a0ba-49f15660d9a0" (UID: "452c96b0-e181-4728-a0ba-49f15660d9a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:53.187510 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.187471 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:53.187510 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.187502 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452c96b0-e181-4728-a0ba-49f15660d9a0-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:53.187510 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.187512 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzdd\" (UniqueName: \"kubernetes.io/projected/452c96b0-e181-4728-a0ba-49f15660d9a0-kube-api-access-xrzdd\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:53.874731 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.874699 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" Nov 03 07:57:53.875107 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.874728 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65xqzn" event={"ID":"452c96b0-e181-4728-a0ba-49f15660d9a0","Type":"ContainerDied","Data":"1c267066fdc9df9f2d17294a04cbe1509ded91b02ec8335fe28f7948eac2db1c"} Nov 03 07:57:53.875107 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:53.874786 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c267066fdc9df9f2d17294a04cbe1509ded91b02ec8335fe28f7948eac2db1c" Nov 03 07:57:54.032041 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.032018 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:54.110918 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.110270 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:54.120851 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.120830 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:54.124672 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.124633 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:54.128927 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.128859 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:54.194830 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.194795 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw\" (UniqueName: \"kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw\") pod \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " Nov 03 07:57:54.195016 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.194853 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v\" (UniqueName: \"kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v\") pod \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " Nov 03 07:57:54.195016 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.194881 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util\") pod \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " Nov 03 07:57:54.195016 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.194927 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle\") pod \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\" (UID: \"396f7685-2ddb-43ce-95b2-30f3b1b8aab2\") " Nov 03 07:57:54.195016 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.194967 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util\") pod \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " Nov 03 07:57:54.195016 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195010 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8\" (UniqueName: \"kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8\") pod \"01f46347-0226-409d-bcf4-fcf4d023763c\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195062 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util\") pod \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195092 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p\" (UniqueName: \"kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p\") pod \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195124 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util\") pod \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195151 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle\") pod \"01f46347-0226-409d-bcf4-fcf4d023763c\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195183 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle\") pod \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\" (UID: \"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195211 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk\" (UniqueName: \"kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk\") pod \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " Nov 03 07:57:54.195282 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195250 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle\") pod \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\" (UID: \"c214afd3-6566-4bb6-b67d-36aa494c2b6a\") " Nov 03 07:57:54.195593 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195293 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util\") pod \"01f46347-0226-409d-bcf4-fcf4d023763c\" (UID: \"01f46347-0226-409d-bcf4-fcf4d023763c\") " Nov 03 07:57:54.195593 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.195334 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle\") pod \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\" (UID: \"2a30be0e-db12-4a4d-b95b-fac92e55ea8d\") " Nov 03 07:57:54.196448 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.196409 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle" pod "2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" (UID: "2fa7fa9a-2d1b-4390-a2c7-ebefd68da915"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.197007 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.196964 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle" pod "c214afd3-6566-4bb6-b67d-36aa494c2b6a" (UID: "c214afd3-6566-4bb6-b67d-36aa494c2b6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.197589 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw" pod "396f7685-2ddb-43ce-95b2-30f3b1b8aab2" (UID: "396f7685-2ddb-43ce-95b2-30f3b1b8aab2"). InnerVolumeSpecName "kube-api-access-kzfgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.197696 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle" pod "01f46347-0226-409d-bcf4-fcf4d023763c" (UID: "01f46347-0226-409d-bcf4-fcf4d023763c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.197924 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v" pod "c214afd3-6566-4bb6-b67d-36aa494c2b6a" (UID: "c214afd3-6566-4bb6-b67d-36aa494c2b6a"). InnerVolumeSpecName "kube-api-access-8g94v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.198177 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle" pod "396f7685-2ddb-43ce-95b2-30f3b1b8aab2" (UID: "396f7685-2ddb-43ce-95b2-30f3b1b8aab2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.199185 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p" pod "2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" (UID: "2fa7fa9a-2d1b-4390-a2c7-ebefd68da915"). InnerVolumeSpecName "kube-api-access-rlh9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.201745 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util" pod "396f7685-2ddb-43ce-95b2-30f3b1b8aab2" (UID: "396f7685-2ddb-43ce-95b2-30f3b1b8aab2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.203555 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util" pod "2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" (UID: "2fa7fa9a-2d1b-4390-a2c7-ebefd68da915"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.204388 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle" pod "2a30be0e-db12-4a4d-b95b-fac92e55ea8d" (UID: "2a30be0e-db12-4a4d-b95b-fac92e55ea8d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.204840 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.204805 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8" pod "01f46347-0226-409d-bcf4-fcf4d023763c" (UID: "01f46347-0226-409d-bcf4-fcf4d023763c"). InnerVolumeSpecName "kube-api-access-49nx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:54.205350 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.204872 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk" pod "2a30be0e-db12-4a4d-b95b-fac92e55ea8d" (UID: "2a30be0e-db12-4a4d-b95b-fac92e55ea8d"). InnerVolumeSpecName "kube-api-access-x9dpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:57:54.205350 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.205267 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util" pod "c214afd3-6566-4bb6-b67d-36aa494c2b6a" (UID: "c214afd3-6566-4bb6-b67d-36aa494c2b6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.207009 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.206983 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util" pod "01f46347-0226-409d-bcf4-fcf4d023763c" (UID: "01f46347-0226-409d-bcf4-fcf4d023763c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.207338 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.207321 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util" pod "2a30be0e-db12-4a4d-b95b-fac92e55ea8d" (UID: "2a30be0e-db12-4a4d-b95b-fac92e55ea8d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:57:54.297102 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297060 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297102 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297095 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297102 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297105 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nx8\" (UniqueName: \"kubernetes.io/projected/01f46347-0226-409d-bcf4-fcf4d023763c-kube-api-access-49nx8\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297114 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297123 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlh9p\" (UniqueName: \"kubernetes.io/projected/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-kube-api-access-rlh9p\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297131 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297139 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297148 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fa7fa9a-2d1b-4390-a2c7-ebefd68da915-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297157 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9dpk\" (UniqueName: \"kubernetes.io/projected/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-kube-api-access-x9dpk\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297165 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c214afd3-6566-4bb6-b67d-36aa494c2b6a-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297173 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f46347-0226-409d-bcf4-fcf4d023763c-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297180 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a30be0e-db12-4a4d-b95b-fac92e55ea8d-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297187 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzfgw\" (UniqueName: \"kubernetes.io/projected/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-kube-api-access-kzfgw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297196 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g94v\" (UniqueName: \"kubernetes.io/projected/c214afd3-6566-4bb6-b67d-36aa494c2b6a-kube-api-access-8g94v\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.297328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.297205 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/396f7685-2ddb-43ce-95b2-30f3b1b8aab2-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:57:54.881272 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.881235 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" event={"ID":"01f46347-0226-409d-bcf4-fcf4d023763c","Type":"ContainerDied","Data":"8c14029775603adb9b050a6690f5cebfa75cace9c8a4348ba7cfd1a1021ad3ba"} Nov 03 07:57:54.881698 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.881276 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kttpk" Nov 03 07:57:54.881698 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.881285 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c14029775603adb9b050a6690f5cebfa75cace9c8a4348ba7cfd1a1021ad3ba" Nov 03 07:57:54.883446 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.883423 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" Nov 03 07:57:54.883588 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.883458 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx25kj" event={"ID":"c214afd3-6566-4bb6-b67d-36aa494c2b6a","Type":"ContainerDied","Data":"87ddaf5521a600c7bb506aa36cddbfa5f23489774ae79fc34e5980e0564da041"} Nov 03 07:57:54.883588 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.883492 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ddaf5521a600c7bb506aa36cddbfa5f23489774ae79fc34e5980e0564da041" Nov 03 07:57:54.885556 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.885536 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" Nov 03 07:57:54.885556 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.885542 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hf7qm" event={"ID":"396f7685-2ddb-43ce-95b2-30f3b1b8aab2","Type":"ContainerDied","Data":"dfc87b0b4e03575bed51efdd1bc85b69e320f462113412b706e19d629b0811bd"} Nov 03 07:57:54.885735 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.885576 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfc87b0b4e03575bed51efdd1bc85b69e320f462113412b706e19d629b0811bd" Nov 03 07:57:54.887825 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.887803 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" event={"ID":"2fa7fa9a-2d1b-4390-a2c7-ebefd68da915","Type":"ContainerDied","Data":"9ad6096a09e6f9df97d0f0af4c00effa722cdfa91ecd1c1a2acfba4ea92e964f"} Nov 03 07:57:54.887929 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.887838 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad6096a09e6f9df97d0f0af4c00effa722cdfa91ecd1c1a2acfba4ea92e964f" Nov 03 07:57:54.887929 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.887878 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2lcj7" Nov 03 07:57:54.889839 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.889809 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" event={"ID":"2a30be0e-db12-4a4d-b95b-fac92e55ea8d","Type":"ContainerDied","Data":"0e6db9f63f8d5dd6d2769375455f40ba5451970ab035ad962f444c5de0293d7e"} Nov 03 07:57:54.889839 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.889840 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6db9f63f8d5dd6d2769375455f40ba5451970ab035ad962f444c5de0293d7e" Nov 03 07:57:54.889980 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:54.889856 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cghg5w" Nov 03 07:57:55.289836 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.289801 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv"] Nov 03 07:57:55.290166 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290150 5951 cpu_manager.go:414] "containerMap: removing container" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="pull" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290169 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="pull" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290178 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="pull" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290183 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="pull" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290191 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="pull" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290196 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="pull" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290202 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="extract" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290207 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="extract" Nov 03 07:57:55.290218 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290216 5951 cpu_manager.go:414] "containerMap: removing container" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290221 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290228 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290233 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290240 5951 cpu_manager.go:414] "containerMap: removing container" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290244 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290251 5951 cpu_manager.go:414] "containerMap: removing container" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290256 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290261 5951 cpu_manager.go:414] "containerMap: removing container" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290265 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290271 5951 cpu_manager.go:414] "containerMap: removing container" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290275 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290287 5951 cpu_manager.go:414] "containerMap: removing container" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290291 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290298 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290302 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290309 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290314 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290320 5951 cpu_manager.go:414] "containerMap: removing container" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290324 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290335 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290339 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290346 5951 cpu_manager.go:414] "containerMap: removing container" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290350 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290355 5951 cpu_manager.go:414] "containerMap: removing container" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290359 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="pull" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290365 5951 cpu_manager.go:414] "containerMap: removing container" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290370 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290383 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290387 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290394 5951 cpu_manager.go:414] "containerMap: removing container" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290398 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="extract" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290406 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="util" Nov 03 07:57:55.290433 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290411 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="util" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290461 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a30be0e-db12-4a4d-b95b-fac92e55ea8d" containerName="extract" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290468 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c3c847-1f11-4c84-b8b2-7a1f9047ed19" containerName="extract" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290476 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f46347-0226-409d-bcf4-fcf4d023763c" containerName="extract" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290482 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c214afd3-6566-4bb6-b67d-36aa494c2b6a" containerName="extract" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290488 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="452c96b0-e181-4728-a0ba-49f15660d9a0" containerName="extract" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290493 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="396f7685-2ddb-43ce-95b2-30f3b1b8aab2" containerName="extract" Nov 03 07:57:55.291405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.290499 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa7fa9a-2d1b-4390-a2c7-ebefd68da915" containerName="extract" Nov 03 07:57:55.295044 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.295027 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.301549 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.301524 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 07:57:55.302358 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.302340 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 07:57:55.303739 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.303710 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 07:57:55.303875 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.303772 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 07:57:55.303875 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.303781 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 07:57:55.303875 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.303820 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cclgr\"" Nov 03 07:57:55.337346 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.337265 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv"] Nov 03 07:57:55.407736 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.407698 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.407736 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.407740 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.408025 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.407820 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvzd\" (UniqueName: \"kubernetes.io/projected/5f79fe08-451b-4072-b949-73db0bcf685c-kube-api-access-hjvzd\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.408025 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.407876 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f79fe08-451b-4072-b949-73db0bcf685c-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.408025 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.407901 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f79fe08-451b-4072-b949-73db0bcf685c-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.408025 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.407921 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509329 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509289 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509329 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509333 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509573 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509363 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvzd\" (UniqueName: \"kubernetes.io/projected/5f79fe08-451b-4072-b949-73db0bcf685c-kube-api-access-hjvzd\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509573 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509394 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f79fe08-451b-4072-b949-73db0bcf685c-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509573 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509417 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f79fe08-451b-4072-b949-73db0bcf685c-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509573 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509445 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.509929 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.509901 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f79fe08-451b-4072-b949-73db0bcf685c-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.511872 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.511845 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f79fe08-451b-4072-b949-73db0bcf685c-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.512089 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.512069 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.512140 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.512068 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.512175 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.512147 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f79fe08-451b-4072-b949-73db0bcf685c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.516818 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.516797 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvzd\" (UniqueName: \"kubernetes.io/projected/5f79fe08-451b-4072-b949-73db0bcf685c-kube-api-access-hjvzd\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-22lvv\" (UID: \"5f79fe08-451b-4072-b949-73db0bcf685c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.613624 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.613538 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:57:55.912475 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:55.912387 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv"] Nov 03 07:57:55.916042 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:57:55.916014 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f79fe08_451b_4072_b949_73db0bcf685c.slice/crio-caedd02a7976493ad2abad60b05507e56fe97759c9a7d3f21cf6347123953db3 WatchSource:0}: Error finding container caedd02a7976493ad2abad60b05507e56fe97759c9a7d3f21cf6347123953db3: Status 404 returned error can't find the container with id caedd02a7976493ad2abad60b05507e56fe97759c9a7d3f21cf6347123953db3 Nov 03 07:57:56.900622 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:56.900564 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" event={"ID":"5f79fe08-451b-4072-b949-73db0bcf685c","Type":"ContainerStarted","Data":"caedd02a7976493ad2abad60b05507e56fe97759c9a7d3f21cf6347123953db3"} Nov 03 07:57:58.914286 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:58.914254 5951 generic.go:356] "Generic (PLEG): container finished" podID="fe19772c-5c94-4231-a786-a2545ef6362b" containerID="149173db7123133164fe656101bf1e205366aab91739e2819620be46cdcd11da" exitCode=0 Nov 03 07:57:58.914699 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:58.914348 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" event={"ID":"fe19772c-5c94-4231-a786-a2545ef6362b","Type":"ContainerDied","Data":"149173db7123133164fe656101bf1e205366aab91739e2819620be46cdcd11da"} Nov 03 07:57:59.921359 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:59.921322 5951 generic.go:356] "Generic (PLEG): container finished" podID="5f79fe08-451b-4072-b949-73db0bcf685c" containerID="4c3949d108f111d73ac180594b3b172ace8ee2359b2062639afcfa2a48ca876c" exitCode=0 Nov 03 07:57:59.921789 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:57:59.921409 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" event={"ID":"5f79fe08-451b-4072-b949-73db0bcf685c","Type":"ContainerDied","Data":"4c3949d108f111d73ac180594b3b172ace8ee2359b2062639afcfa2a48ca876c"} Nov 03 07:58:00.042818 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.042796 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:58:00.154080 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.154048 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util\") pod \"fe19772c-5c94-4231-a786-a2545ef6362b\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " Nov 03 07:58:00.154252 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.154134 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle\") pod \"fe19772c-5c94-4231-a786-a2545ef6362b\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " Nov 03 07:58:00.154252 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.154167 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq\" (UniqueName: \"kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq\") pod \"fe19772c-5c94-4231-a786-a2545ef6362b\" (UID: \"fe19772c-5c94-4231-a786-a2545ef6362b\") " Nov 03 07:58:00.154568 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.154544 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle" pod "fe19772c-5c94-4231-a786-a2545ef6362b" (UID: "fe19772c-5c94-4231-a786-a2545ef6362b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:00.156388 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.156363 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq" pod "fe19772c-5c94-4231-a786-a2545ef6362b" (UID: "fe19772c-5c94-4231-a786-a2545ef6362b"). InnerVolumeSpecName "kube-api-access-d5ksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:58:00.159255 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.159231 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util" pod "fe19772c-5c94-4231-a786-a2545ef6362b" (UID: "fe19772c-5c94-4231-a786-a2545ef6362b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:00.255267 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.255190 5951 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-util\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:58:00.255267 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.255216 5951 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe19772c-5c94-4231-a786-a2545ef6362b-bundle\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:58:00.255267 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.255229 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ksq\" (UniqueName: \"kubernetes.io/projected/fe19772c-5c94-4231-a786-a2545ef6362b-kube-api-access-d5ksq\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:58:00.927263 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.927228 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" event={"ID":"fe19772c-5c94-4231-a786-a2545ef6362b","Type":"ContainerDied","Data":"ed9adaaf3fd0fc65558965b9cf9cacc8efc6205a5284540966333135d6c4bbf2"} Nov 03 07:58:00.927678 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.927278 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9adaaf3fd0fc65558965b9cf9cacc8efc6205a5284540966333135d6c4bbf2" Nov 03 07:58:00.927678 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.927244 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6m4xqh" Nov 03 07:58:00.929529 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.929500 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" event={"ID":"5f79fe08-451b-4072-b949-73db0bcf685c","Type":"ContainerStarted","Data":"6ed2a8556d4ac5272a1c22f2b9c2616b5284d4f388f076f1984a4414d2325134"} Nov 03 07:58:00.929699 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.929682 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:58:00.970185 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:00.970127 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" podStartSLOduration=1.8049769919999998 podStartE2EDuration="5.970112268s" totalImagesPullingTime="3.67025088s" totalInitContainerRuntime="494.884396ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:00.968631825 +0000 UTC m=+220.350082742" watchObservedRunningTime="2026-11-03 07:58:00.970112268 +0000 UTC m=+220.351563198" Nov 03 07:58:01.954033 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:58:01.953998 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:58:02.807555 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807516 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd"] Nov 03 07:58:02.807841 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807828 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="util" Nov 03 07:58:02.807886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807842 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="util" Nov 03 07:58:02.807886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807857 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="extract" Nov 03 07:58:02.807886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807863 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="extract" Nov 03 07:58:02.807886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807870 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="pull" Nov 03 07:58:02.807886 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807875 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="pull" Nov 03 07:58:02.808039 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.807928 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe19772c-5c94-4231-a786-a2545ef6362b" containerName="extract" Nov 03 07:58:02.810537 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.810517 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:02.813501 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.813479 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:58:02.813603 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.813510 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 07:58:02.813603 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.813564 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-n57jb\"" Nov 03 07:58:02.822242 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.822219 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd"] Nov 03 07:58:02.978126 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.978085 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b173126-a69a-40c0-bf53-3f1706ea49be-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd\" (UID: \"0b173126-a69a-40c0-bf53-3f1706ea49be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:02.978519 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:02.978133 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzdg\" (UniqueName: \"kubernetes.io/projected/0b173126-a69a-40c0-bf53-3f1706ea49be-kube-api-access-ppzdg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd\" (UID: \"0b173126-a69a-40c0-bf53-3f1706ea49be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:03.079038 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.078953 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b173126-a69a-40c0-bf53-3f1706ea49be-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd\" (UID: \"0b173126-a69a-40c0-bf53-3f1706ea49be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:03.079038 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.078992 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzdg\" (UniqueName: \"kubernetes.io/projected/0b173126-a69a-40c0-bf53-3f1706ea49be-kube-api-access-ppzdg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd\" (UID: \"0b173126-a69a-40c0-bf53-3f1706ea49be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:03.079396 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.079373 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b173126-a69a-40c0-bf53-3f1706ea49be-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd\" (UID: \"0b173126-a69a-40c0-bf53-3f1706ea49be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:03.094664 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.094621 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzdg\" (UniqueName: \"kubernetes.io/projected/0b173126-a69a-40c0-bf53-3f1706ea49be-kube-api-access-ppzdg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd\" (UID: \"0b173126-a69a-40c0-bf53-3f1706ea49be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:03.122268 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.122240 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" Nov 03 07:58:03.402111 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:03.402069 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b173126_a69a_40c0_bf53_3f1706ea49be.slice/crio-fc84aad89e5501423e63405d2fd1af232f9b525f43882d9457e04941b985374c WatchSource:0}: Error finding container fc84aad89e5501423e63405d2fd1af232f9b525f43882d9457e04941b985374c: Status 404 returned error can't find the container with id fc84aad89e5501423e63405d2fd1af232f9b525f43882d9457e04941b985374c Nov 03 07:58:03.410930 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.410894 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd"] Nov 03 07:58:03.943400 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:03.943366 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" event={"ID":"0b173126-a69a-40c0-bf53-3f1706ea49be","Type":"ContainerStarted","Data":"fc84aad89e5501423e63405d2fd1af232f9b525f43882d9457e04941b985374c"} Nov 03 07:58:08.465614 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.465576 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4"] Nov 03 07:58:08.468964 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.468948 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.471959 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.471931 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 07:58:08.472183 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.472167 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 07:58:08.472247 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.472198 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-sk5sb\"" Nov 03 07:58:08.476562 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.476544 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4"] Nov 03 07:58:08.629451 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.629408 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhk8\" (UniqueName: \"kubernetes.io/projected/6f99ae79-335d-4930-bb13-5cfe8914fa86-kube-api-access-snhk8\") pod \"jobset-operator-674cd59dc5-zn6l4\" (UID: \"6f99ae79-335d-4930-bb13-5cfe8914fa86\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.629637 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.629474 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f99ae79-335d-4930-bb13-5cfe8914fa86-tmp\") pod \"jobset-operator-674cd59dc5-zn6l4\" (UID: \"6f99ae79-335d-4930-bb13-5cfe8914fa86\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.730793 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.730728 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhk8\" (UniqueName: \"kubernetes.io/projected/6f99ae79-335d-4930-bb13-5cfe8914fa86-kube-api-access-snhk8\") pod \"jobset-operator-674cd59dc5-zn6l4\" (UID: \"6f99ae79-335d-4930-bb13-5cfe8914fa86\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.730793 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.730779 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f99ae79-335d-4930-bb13-5cfe8914fa86-tmp\") pod \"jobset-operator-674cd59dc5-zn6l4\" (UID: \"6f99ae79-335d-4930-bb13-5cfe8914fa86\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.731126 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.731107 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f99ae79-335d-4930-bb13-5cfe8914fa86-tmp\") pod \"jobset-operator-674cd59dc5-zn6l4\" (UID: \"6f99ae79-335d-4930-bb13-5cfe8914fa86\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.738300 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.738276 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhk8\" (UniqueName: \"kubernetes.io/projected/6f99ae79-335d-4930-bb13-5cfe8914fa86-kube-api-access-snhk8\") pod \"jobset-operator-674cd59dc5-zn6l4\" (UID: \"6f99ae79-335d-4930-bb13-5cfe8914fa86\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:08.788120 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:08.788095 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" Nov 03 07:58:09.063194 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:09.063164 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4"] Nov 03 07:58:09.066567 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:09.066533 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f99ae79_335d_4930_bb13_5cfe8914fa86.slice/crio-8da9e244773a4d025d55057de6a411c4205db7803731e99010c7335a78680329 WatchSource:0}: Error finding container 8da9e244773a4d025d55057de6a411c4205db7803731e99010c7335a78680329: Status 404 returned error can't find the container with id 8da9e244773a4d025d55057de6a411c4205db7803731e99010c7335a78680329 Nov 03 07:58:09.970791 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:09.970741 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" event={"ID":"6f99ae79-335d-4930-bb13-5cfe8914fa86","Type":"ContainerStarted","Data":"8da9e244773a4d025d55057de6a411c4205db7803731e99010c7335a78680329"} Nov 03 07:58:10.976835 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:10.976794 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" event={"ID":"0b173126-a69a-40c0-bf53-3f1706ea49be","Type":"ContainerStarted","Data":"62cfa7ba556bd0ebded8dd9c18d2e6a352fb5dd222d7f780255700cf50a8d5df"} Nov 03 07:58:10.998101 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:10.998048 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dhjtd" podStartSLOduration=1.556897242 podStartE2EDuration="8.998030962s" totalImagesPullingTime="7.44113372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:10.996072807 +0000 UTC m=+230.377523724" watchObservedRunningTime="2026-11-03 07:58:10.998030962 +0000 UTC m=+230.379481883" Nov 03 07:58:11.935956 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:11.935922 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-22lvv" Nov 03 07:58:17.005947 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:17.005912 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" event={"ID":"6f99ae79-335d-4930-bb13-5cfe8914fa86","Type":"ContainerStarted","Data":"2081dd4b66cf7d90fdd3d7786a3dfaea1ae7218963173d4fb2cd4e6225136142"} Nov 03 07:58:17.024344 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:17.024293 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zn6l4" podStartSLOduration=2.102140008 podStartE2EDuration="9.024275745s" totalImagesPullingTime="6.922135737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:17.022627177 +0000 UTC m=+236.404078106" watchObservedRunningTime="2026-11-03 07:58:17.024275745 +0000 UTC m=+236.405726662" Nov 03 07:58:18.251386 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.251354 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc"] Nov 03 07:58:18.254909 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.254885 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.258188 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.258165 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 07:58:18.258280 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.258237 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 07:58:18.258280 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.258257 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7wkkr\"" Nov 03 07:58:18.263205 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.263184 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc"] Nov 03 07:58:18.320044 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.320017 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-59lkc\" (UID: \"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.320173 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.320055 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkhz\" (UniqueName: \"kubernetes.io/projected/efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c-kube-api-access-9qkhz\") pod \"cert-manager-webhook-5dbbbf75cc-59lkc\" (UID: \"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.421001 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.420971 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkhz\" (UniqueName: \"kubernetes.io/projected/efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c-kube-api-access-9qkhz\") pod \"cert-manager-webhook-5dbbbf75cc-59lkc\" (UID: \"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.421179 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.421041 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-59lkc\" (UID: \"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.428290 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.428258 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-59lkc\" (UID: \"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.428388 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.428272 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkhz\" (UniqueName: \"kubernetes.io/projected/efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c-kube-api-access-9qkhz\") pod \"cert-manager-webhook-5dbbbf75cc-59lkc\" (UID: \"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.567362 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.567333 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:18.838559 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:18.838433 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc"] Nov 03 07:58:18.840855 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:18.840826 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefcd3e0b_6ce9_4cbb_bf92_cad5f81b7b1c.slice/crio-9692a204b6a8465c460073f161b6b514a497144a55f83a02f73f419e69e03ad1 WatchSource:0}: Error finding container 9692a204b6a8465c460073f161b6b514a497144a55f83a02f73f419e69e03ad1: Status 404 returned error can't find the container with id 9692a204b6a8465c460073f161b6b514a497144a55f83a02f73f419e69e03ad1 Nov 03 07:58:19.015893 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:19.015858 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" event={"ID":"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c","Type":"ContainerStarted","Data":"9692a204b6a8465c460073f161b6b514a497144a55f83a02f73f419e69e03ad1"} Nov 03 07:58:20.914019 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.913985 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw"] Nov 03 07:58:20.917317 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.917299 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:20.923906 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.923885 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x727v\"" Nov 03 07:58:20.924021 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.923930 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 07:58:20.924021 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.924002 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 07:58:20.929924 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.929901 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw"] Nov 03 07:58:20.943459 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:20.943427 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbsnr\" (UniqueName: \"kubernetes.io/projected/c9136285-1045-4ffc-9591-09176f766fd8-kube-api-access-lbsnr\") pod \"limitador-operator-controller-manager-5778bb4449-dzbtw\" (UID: \"c9136285-1045-4ffc-9591-09176f766fd8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:21.044786 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:21.044756 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbsnr\" (UniqueName: \"kubernetes.io/projected/c9136285-1045-4ffc-9591-09176f766fd8-kube-api-access-lbsnr\") pod \"limitador-operator-controller-manager-5778bb4449-dzbtw\" (UID: \"c9136285-1045-4ffc-9591-09176f766fd8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:21.058094 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:21.058060 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbsnr\" (UniqueName: \"kubernetes.io/projected/c9136285-1045-4ffc-9591-09176f766fd8-kube-api-access-lbsnr\") pod \"limitador-operator-controller-manager-5778bb4449-dzbtw\" (UID: \"c9136285-1045-4ffc-9591-09176f766fd8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:21.230865 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:21.230779 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:21.514901 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:21.514873 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw"] Nov 03 07:58:21.517307 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:21.517275 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9136285_1045_4ffc_9591_09176f766fd8.slice/crio-fc91dc6f610aa9fe7d6f5d661c2f3455b9391d0bd6824b86187a1be95935f1ef WatchSource:0}: Error finding container fc91dc6f610aa9fe7d6f5d661c2f3455b9391d0bd6824b86187a1be95935f1ef: Status 404 returned error can't find the container with id fc91dc6f610aa9fe7d6f5d661c2f3455b9391d0bd6824b86187a1be95935f1ef Nov 03 07:58:22.033297 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:22.033258 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" event={"ID":"c9136285-1045-4ffc-9591-09176f766fd8","Type":"ContainerStarted","Data":"fc91dc6f610aa9fe7d6f5d661c2f3455b9391d0bd6824b86187a1be95935f1ef"} Nov 03 07:58:23.041461 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:23.041228 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" event={"ID":"efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c","Type":"ContainerStarted","Data":"d1d2fcc9ca3f303a6499f41893132598977ba03b26f7da0ff963e6f804bce426"} Nov 03 07:58:23.041461 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:23.041395 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:23.061681 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:23.060886 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podStartSLOduration=1.475216452 podStartE2EDuration="5.060867352s" totalImagesPullingTime="3.5856509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:23.059078603 +0000 UTC m=+242.440529532" watchObservedRunningTime="2026-11-03 07:58:23.060867352 +0000 UTC m=+242.442318267" Nov 03 07:58:25.052409 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:25.052375 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" event={"ID":"c9136285-1045-4ffc-9591-09176f766fd8","Type":"ContainerStarted","Data":"9d9ccce6a4978231ff7b62808084ba243ce994a3f9cbd8fa92071e3149056928"} Nov 03 07:58:25.052827 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:25.052463 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:25.068634 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:25.068577 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" podStartSLOduration=1.754403505 podStartE2EDuration="5.068560727s" totalImagesPullingTime="3.314157222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:25.067777599 +0000 UTC m=+244.449228515" watchObservedRunningTime="2026-11-03 07:58:25.068560727 +0000 UTC m=+244.450011642" Nov 03 07:58:26.996971 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:26.996932 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz"] Nov 03 07:58:27.000618 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.000595 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.006324 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.006302 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f6gvb\"" Nov 03 07:58:27.011112 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.011079 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz"] Nov 03 07:58:27.104173 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.104137 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsp5\" (UniqueName: \"kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pvwzz\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.104320 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.104198 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pvwzz\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.205358 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.205308 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsp5\" (UniqueName: \"kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pvwzz\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.205503 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.205363 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pvwzz\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.205763 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.205743 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pvwzz\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.213050 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.213024 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsp5\" (UniqueName: \"kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-pvwzz\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.313583 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.313555 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:27.604573 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:27.604546 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz"] Nov 03 07:58:27.606233 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:27.606205 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b8fc8_d752_4c4f_a156_0031f6954908.slice/crio-7aa847663d7e82089893d1f965fee429b35bd7118dbca344e2bd1189112d316c WatchSource:0}: Error finding container 7aa847663d7e82089893d1f965fee429b35bd7118dbca344e2bd1189112d316c: Status 404 returned error can't find the container with id 7aa847663d7e82089893d1f965fee429b35bd7118dbca344e2bd1189112d316c Nov 03 07:58:28.067318 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:28.067289 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" event={"ID":"b61b8fc8-d752-4c4f-a156-0031f6954908","Type":"ContainerStarted","Data":"7aa847663d7e82089893d1f965fee429b35bd7118dbca344e2bd1189112d316c"} Nov 03 07:58:29.048652 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:29.048621 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" Nov 03 07:58:31.953790 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:58:31.953750 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:58:32.012381 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.012340 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5"] Nov 03 07:58:32.019986 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.019961 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.030157 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.030136 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jqwzn\"" Nov 03 07:58:32.034065 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.034034 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5"] Nov 03 07:58:32.149857 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.149823 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105f3437-0d08-43f6-a573-248cd4f36a96-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150015 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.149881 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150015 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.149962 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150090 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.150028 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150124 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.150096 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150157 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.150119 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150157 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.150138 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105f3437-0d08-43f6-a573-248cd4f36a96-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150270 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.150250 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105f3437-0d08-43f6-a573-248cd4f36a96-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.150328 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.150280 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v99f\" (UniqueName: \"kubernetes.io/projected/105f3437-0d08-43f6-a573-248cd4f36a96-kube-api-access-8v99f\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251170 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251230 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251258 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251257 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251544 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251280 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105f3437-0d08-43f6-a573-248cd4f36a96-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251544 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251336 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105f3437-0d08-43f6-a573-248cd4f36a96-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251544 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251359 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v99f\" (UniqueName: \"kubernetes.io/projected/105f3437-0d08-43f6-a573-248cd4f36a96-kube-api-access-8v99f\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251544 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251393 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105f3437-0d08-43f6-a573-248cd4f36a96-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251544 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251439 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251544 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251475 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.251865 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.251622 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.252164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.252135 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.252414 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.252393 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.252810 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.252785 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.253092 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.253073 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105f3437-0d08-43f6-a573-248cd4f36a96-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.254307 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.254287 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105f3437-0d08-43f6-a573-248cd4f36a96-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.254677 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.254656 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105f3437-0d08-43f6-a573-248cd4f36a96-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.260701 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.260637 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105f3437-0d08-43f6-a573-248cd4f36a96-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.262608 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.262496 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v99f\" (UniqueName: \"kubernetes.io/projected/105f3437-0d08-43f6-a573-248cd4f36a96-kube-api-access-8v99f\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-nzds5\" (UID: \"105f3437-0d08-43f6-a573-248cd4f36a96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.336460 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.336412 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:32.630475 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.630449 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5"] Nov 03 07:58:32.632535 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:32.632494 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105f3437_0d08_43f6_a573_248cd4f36a96.slice/crio-a78abf47a25f8b899fd7377c95b7f015a542a7e568812c54de8cf07ce5e0cfea WatchSource:0}: Error finding container a78abf47a25f8b899fd7377c95b7f015a542a7e568812c54de8cf07ce5e0cfea: Status 404 returned error can't find the container with id a78abf47a25f8b899fd7377c95b7f015a542a7e568812c54de8cf07ce5e0cfea Nov 03 07:58:32.635100 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.635070 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:58:32.635167 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.635140 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:58:32.635208 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:32.635166 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:58:33.097502 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:33.097460 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" event={"ID":"105f3437-0d08-43f6-a573-248cd4f36a96","Type":"ContainerStarted","Data":"093cb1d70d05e740ad5e73b37eb3413e2ec94e9097006ff44aa5c3bb916fe01d"} Nov 03 07:58:33.097924 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:33.097531 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" event={"ID":"105f3437-0d08-43f6-a573-248cd4f36a96","Type":"ContainerStarted","Data":"a78abf47a25f8b899fd7377c95b7f015a542a7e568812c54de8cf07ce5e0cfea"} Nov 03 07:58:33.122127 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:33.121724 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" podStartSLOduration=2.121706172 podStartE2EDuration="2.121706172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:33.121050965 +0000 UTC m=+252.502501892" watchObservedRunningTime="2026-11-03 07:58:33.121706172 +0000 UTC m=+252.503157091" Nov 03 07:58:33.337028 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:33.336988 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:33.342966 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:33.342927 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:34.103118 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:34.103082 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:34.104341 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:34.104314 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-nzds5" Nov 03 07:58:35.109891 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:35.109854 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" event={"ID":"b61b8fc8-d752-4c4f-a156-0031f6954908","Type":"ContainerStarted","Data":"8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22"} Nov 03 07:58:35.110324 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:35.110144 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:35.129245 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:35.129196 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" podStartSLOduration=2.219287125 podStartE2EDuration="9.129180069s" totalImagesPullingTime="6.909892944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:35.128124005 +0000 UTC m=+254.509574946" watchObservedRunningTime="2026-11-03 07:58:35.129180069 +0000 UTC m=+254.510630989" Nov 03 07:58:36.060156 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:36.060120 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dzbtw" Nov 03 07:58:46.116848 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.116816 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:46.503054 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.502980 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth"] Nov 03 07:58:46.508728 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.508710 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.521450 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.521424 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth"] Nov 03 07:58:46.677361 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.677332 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89psz\" (UniqueName: \"kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz\") pod \"kuadrant-operator-controller-manager-678d87766f-q9cth\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.677532 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.677385 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q9cth\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.778479 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.778452 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89psz\" (UniqueName: \"kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz\") pod \"kuadrant-operator-controller-manager-678d87766f-q9cth\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.778632 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.778507 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q9cth\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.778887 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.778872 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q9cth\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.787007 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.786983 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89psz\" (UniqueName: \"kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz\") pod \"kuadrant-operator-controller-manager-678d87766f-q9cth\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:46.821657 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:46.821623 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:47.111170 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:47.111147 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth"] Nov 03 07:58:47.113663 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:47.113620 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ec4ebb_9864_4318_bbc4_39bef44cac21.slice/crio-d568cb9d777bc2a5b69bd75b219f77215f7053a014412d0087b9de27ff28d0f3 WatchSource:0}: Error finding container d568cb9d777bc2a5b69bd75b219f77215f7053a014412d0087b9de27ff28d0f3: Status 404 returned error can't find the container with id d568cb9d777bc2a5b69bd75b219f77215f7053a014412d0087b9de27ff28d0f3 Nov 03 07:58:47.167389 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:47.167363 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" event={"ID":"54ec4ebb-9864-4318-bbc4-39bef44cac21","Type":"ContainerStarted","Data":"d568cb9d777bc2a5b69bd75b219f77215f7053a014412d0087b9de27ff28d0f3"} Nov 03 07:58:48.173162 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:48.173126 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" event={"ID":"54ec4ebb-9864-4318-bbc4-39bef44cac21","Type":"ContainerStarted","Data":"35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392"} Nov 03 07:58:48.173554 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:48.173196 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:48.192306 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:48.192255 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" podStartSLOduration=2.192238199 podStartE2EDuration="2.192238199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:48.191229729 +0000 UTC m=+267.572680670" watchObservedRunningTime="2026-11-03 07:58:48.192238199 +0000 UTC m=+267.573689117" Nov 03 07:58:53.948073 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.948038 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm"] Nov 03 07:58:53.977085 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.977043 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm"] Nov 03 07:58:53.977259 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.977159 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:53.980180 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.980121 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-t257d\"" Nov 03 07:58:53.980180 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.980142 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 07:58:53.980180 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.980176 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 07:58:53.980357 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:53.980201 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 07:58:54.145621 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.145588 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/9777f6de-530f-40d3-88e9-ba27fc7b1b85-kube-api-access-5vl8p\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.145797 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.145651 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9777f6de-530f-40d3-88e9-ba27fc7b1b85-manager-config\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.145797 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.145752 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9777f6de-530f-40d3-88e9-ba27fc7b1b85-cert\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.145871 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.145844 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9777f6de-530f-40d3-88e9-ba27fc7b1b85-metrics-certs\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.246591 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.246520 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9777f6de-530f-40d3-88e9-ba27fc7b1b85-cert\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.246591 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.246578 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9777f6de-530f-40d3-88e9-ba27fc7b1b85-metrics-certs\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.246788 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.246751 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/9777f6de-530f-40d3-88e9-ba27fc7b1b85-kube-api-access-5vl8p\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.246836 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.246800 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9777f6de-530f-40d3-88e9-ba27fc7b1b85-manager-config\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.247415 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.247387 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9777f6de-530f-40d3-88e9-ba27fc7b1b85-manager-config\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.249208 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.249181 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9777f6de-530f-40d3-88e9-ba27fc7b1b85-metrics-certs\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.249324 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.249215 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9777f6de-530f-40d3-88e9-ba27fc7b1b85-cert\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.257908 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.257887 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/9777f6de-530f-40d3-88e9-ba27fc7b1b85-kube-api-access-5vl8p\") pod \"jobset-controller-manager-5499fb4776-qx4xm\" (UID: \"9777f6de-530f-40d3-88e9-ba27fc7b1b85\") " pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.305976 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.305944 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:54.595390 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:54.595362 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm"] Nov 03 07:58:54.597297 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:58:54.597268 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9777f6de_530f_40d3_88e9_ba27fc7b1b85.slice/crio-37977f0c86d4e48fa512fd7b1165221a717940c5aa2aa8e4f58436c9710c29ee WatchSource:0}: Error finding container 37977f0c86d4e48fa512fd7b1165221a717940c5aa2aa8e4f58436c9710c29ee: Status 404 returned error can't find the container with id 37977f0c86d4e48fa512fd7b1165221a717940c5aa2aa8e4f58436c9710c29ee Nov 03 07:58:55.207021 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:55.206987 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" event={"ID":"9777f6de-530f-40d3-88e9-ba27fc7b1b85","Type":"ContainerStarted","Data":"37977f0c86d4e48fa512fd7b1165221a717940c5aa2aa8e4f58436c9710c29ee"} Nov 03 07:58:59.180027 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.179992 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 07:58:59.231753 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.231710 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" event={"ID":"9777f6de-530f-40d3-88e9-ba27fc7b1b85","Type":"ContainerStarted","Data":"5ee45a15bd74f79a38c21b571aabb54654f9d729f1121a8c0e5e66b24dcba0a4"} Nov 03 07:58:59.231903 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.231840 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:58:59.318450 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.318403 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" podStartSLOduration=2.732179914 podStartE2EDuration="6.318389089s" totalImagesPullingTime="3.586209175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:59.31652458 +0000 UTC m=+278.697975499" watchObservedRunningTime="2026-11-03 07:58:59.318389089 +0000 UTC m=+278.699840006" Nov 03 07:58:59.340865 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.340835 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz"] Nov 03 07:58:59.341124 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.341093 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" podUID="b61b8fc8-d752-4c4f-a156-0031f6954908" containerName="manager" containerID="cri-o://8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22" gracePeriod=10 Nov 03 07:58:59.582735 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.582708 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:58:59.603240 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.603214 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume\") pod \"b61b8fc8-d752-4c4f-a156-0031f6954908\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " Nov 03 07:58:59.603373 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.603246 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5\" (UniqueName: \"kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5\") pod \"b61b8fc8-d752-4c4f-a156-0031f6954908\" (UID: \"b61b8fc8-d752-4c4f-a156-0031f6954908\") " Nov 03 07:58:59.603732 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.603706 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume" pod "b61b8fc8-d752-4c4f-a156-0031f6954908" (UID: "b61b8fc8-d752-4c4f-a156-0031f6954908"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:58:59.605438 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.605413 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5" pod "b61b8fc8-d752-4c4f-a156-0031f6954908" (UID: "b61b8fc8-d752-4c4f-a156-0031f6954908"). InnerVolumeSpecName "kube-api-access-swsp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:58:59.703921 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.703888 5951 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b61b8fc8-d752-4c4f-a156-0031f6954908-extensions-socket-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:58:59.703921 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:58:59.703914 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swsp5\" (UniqueName: \"kubernetes.io/projected/b61b8fc8-d752-4c4f-a156-0031f6954908-kube-api-access-swsp5\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:59:00.237624 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.237585 5951 generic.go:356] "Generic (PLEG): container finished" podID="b61b8fc8-d752-4c4f-a156-0031f6954908" containerID="8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22" exitCode=0 Nov 03 07:59:00.237624 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.237630 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" Nov 03 07:59:00.238189 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.237677 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" event={"ID":"b61b8fc8-d752-4c4f-a156-0031f6954908","Type":"ContainerDied","Data":"8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22"} Nov 03 07:59:00.238189 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.237736 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz" event={"ID":"b61b8fc8-d752-4c4f-a156-0031f6954908","Type":"ContainerDied","Data":"7aa847663d7e82089893d1f965fee429b35bd7118dbca344e2bd1189112d316c"} Nov 03 07:59:00.238189 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.237768 5951 scope.go:122] "RemoveContainer" containerID="8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22" Nov 03 07:59:00.246837 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.246817 5951 scope.go:122] "RemoveContainer" containerID="8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22" Nov 03 07:59:00.247064 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:59:00.247047 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22\": container with ID starting with 8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22 not found: ID does not exist" containerID="8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22" Nov 03 07:59:00.247109 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.247071 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22"} err="failed to get container status \"8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22\": rpc error: code = NotFound desc = could not find container \"8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22\": container with ID starting with 8dc8cff1950ab51c9823bc6ba0bfdf49c15f34eb9f4a209d63f925da67b3bf22 not found: ID does not exist" Nov 03 07:59:00.260969 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.260948 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz"] Nov 03 07:59:00.266951 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.266932 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-pvwzz"] Nov 03 07:59:00.734755 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:00.734725 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61b8fc8-d752-4c4f-a156-0031f6954908" path="/var/lib/kubelet/pods/b61b8fc8-d752-4c4f-a156-0031f6954908/volumes" Nov 03 07:59:01.953854 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:59:01.953816 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:59:10.243389 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:10.243356 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5499fb4776-qx4xm" Nov 03 07:59:20.777996 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:20.777968 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:59:20.780117 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:20.780100 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 07:59:27.620863 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.620829 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-8kjgc"] Nov 03 07:59:27.621233 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.621192 5951 cpu_manager.go:414] "containerMap: removing container" podUID="b61b8fc8-d752-4c4f-a156-0031f6954908" containerName="manager" Nov 03 07:59:27.621233 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.621203 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61b8fc8-d752-4c4f-a156-0031f6954908" containerName="manager" Nov 03 07:59:27.621306 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.621289 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61b8fc8-d752-4c4f-a156-0031f6954908" containerName="manager" Nov 03 07:59:27.624107 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.624085 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.638671 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.638616 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8kjgc"] Nov 03 07:59:27.664106 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.664075 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 07:59:27.664106 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.664075 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 07:59:27.682989 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.682956 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 07:59:27.683150 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.682998 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wdxdf\"" Nov 03 07:59:27.683150 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.683101 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 07:59:27.745704 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.745666 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbnw\" (UniqueName: \"kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.745895 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.745753 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.745895 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.745813 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.846860 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.846828 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.847065 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.846898 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbnw\" (UniqueName: \"kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.847065 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.846994 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.847223 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:59:27.847133 5951 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found Nov 03 07:59:27.847223 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:59:27.847221 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs podName:297faaa7-4afc-4bfb-b62c-251b209be679 nodeName:}" failed. No retries permitted until 2026-11-03 07:59:28.34719935 +0000 UTC m=+307.728650245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs") pod "odh-model-controller-769875db56-8kjgc" (UID: "297faaa7-4afc-4bfb-b62c-251b209be679") : secret "odh-model-controller-metrics-tls" not found Nov 03 07:59:27.849880 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.849853 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:27.872046 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:27.871972 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbnw\" (UniqueName: \"kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:28.351320 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:28.351280 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:28.353838 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:28.353819 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\") pod \"odh-model-controller-769875db56-8kjgc\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:28.536476 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:28.536440 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:28.812067 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:28.812040 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8kjgc"] Nov 03 07:59:28.814545 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:59:28.814519 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297faaa7_4afc_4bfb_b62c_251b209be679.slice/crio-b70f52d2f026287c9810a395e928547729aff9fb41d70d54212adb5b4b62c99b WatchSource:0}: Error finding container b70f52d2f026287c9810a395e928547729aff9fb41d70d54212adb5b4b62c99b: Status 404 returned error can't find the container with id b70f52d2f026287c9810a395e928547729aff9fb41d70d54212adb5b4b62c99b Nov 03 07:59:29.390714 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:29.390674 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8kjgc" event={"ID":"297faaa7-4afc-4bfb-b62c-251b209be679","Type":"ContainerStarted","Data":"b70f52d2f026287c9810a395e928547729aff9fb41d70d54212adb5b4b62c99b"} Nov 03 07:59:31.954827 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:59:31.954787 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 07:59:33.417976 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:33.417925 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8kjgc" event={"ID":"297faaa7-4afc-4bfb-b62c-251b209be679","Type":"ContainerStarted","Data":"8ba73ca46313e24e19b59342b788a31c3d4995f571f59ade4729cd83553de0bb"} Nov 03 07:59:33.418480 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:33.418030 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:33.432703 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:33.432626 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-8kjgc" podStartSLOduration=2.044570591 podStartE2EDuration="6.432603706s" totalImagesPullingTime="4.388033115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:33.431475253 +0000 UTC m=+312.812926260" watchObservedRunningTime="2026-11-03 07:59:33.432603706 +0000 UTC m=+312.814054623" Nov 03 07:59:41.590326 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.590297 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ctmbl"] Nov 03 07:59:41.593785 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.593764 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.596548 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.596520 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 07:59:41.596686 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.596552 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k97gv\"" Nov 03 07:59:41.596686 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.596588 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 07:59:41.596686 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.596602 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 07:59:41.602590 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.602554 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ctmbl"] Nov 03 07:59:41.775123 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.775091 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.775123 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.775126 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.775330 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.775161 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8th2d\" (UniqueName: \"kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.875712 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.875601 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.875712 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.875682 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.875944 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.875726 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8th2d\" (UniqueName: \"kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.878303 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.878279 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.878464 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.878353 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.885589 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.885560 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8th2d\" (UniqueName: \"kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d\") pod \"maas-controller-579cf7c6d-ctmbl\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.907553 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.907521 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:41.908135 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.908115 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-799454fcf4-24ntl"] Nov 03 07:59:41.911352 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.911335 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:41.921626 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:41.921603 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799454fcf4-24ntl"] Nov 03 07:59:42.076882 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.076843 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.077063 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.076892 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.077063 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.076913 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4r7\" (UniqueName: \"kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.178538 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.178463 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.178538 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.178517 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.178769 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.178691 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4r7\" (UniqueName: \"kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.181073 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.181049 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.181305 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.181282 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.185382 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.185358 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ctmbl"] Nov 03 07:59:42.187715 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:59:42.187685 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f89838_58cc_491f_9224_21503b685ffd.slice/crio-57c1aeb9897e96fbcbe11ba33096ffae6a9ad0268da9dfda7593dc7a251d74d7 WatchSource:0}: Error finding container 57c1aeb9897e96fbcbe11ba33096ffae6a9ad0268da9dfda7593dc7a251d74d7: Status 404 returned error can't find the container with id 57c1aeb9897e96fbcbe11ba33096ffae6a9ad0268da9dfda7593dc7a251d74d7 Nov 03 07:59:42.187828 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.187743 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4r7\" (UniqueName: \"kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7\") pod \"maas-controller-799454fcf4-24ntl\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.244996 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.244967 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:42.463955 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.463869 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" event={"ID":"85f89838-58cc-491f-9224-21503b685ffd","Type":"ContainerStarted","Data":"57c1aeb9897e96fbcbe11ba33096ffae6a9ad0268da9dfda7593dc7a251d74d7"} Nov 03 07:59:42.515405 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:42.515378 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799454fcf4-24ntl"] Nov 03 07:59:42.517505 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:59:42.517476 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b25251_3fe9_4716_b419_6d13474804f6.slice/crio-ff624d8e9c2a44dddfe12597ff493f03ee00f566314b812f0c3234ea9abd759b WatchSource:0}: Error finding container ff624d8e9c2a44dddfe12597ff493f03ee00f566314b812f0c3234ea9abd759b: Status 404 returned error can't find the container with id ff624d8e9c2a44dddfe12597ff493f03ee00f566314b812f0c3234ea9abd759b Nov 03 07:59:43.471164 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:43.471122 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-24ntl" event={"ID":"66b25251-3fe9-4716-b419-6d13474804f6","Type":"ContainerStarted","Data":"ff624d8e9c2a44dddfe12597ff493f03ee00f566314b812f0c3234ea9abd759b"} Nov 03 07:59:44.427228 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:44.427194 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 07:59:46.490949 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:46.490845 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-24ntl" event={"ID":"66b25251-3fe9-4716-b419-6d13474804f6","Type":"ContainerStarted","Data":"dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53"} Nov 03 07:59:46.490949 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:46.490919 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:46.492634 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:46.492598 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" event={"ID":"85f89838-58cc-491f-9224-21503b685ffd","Type":"ContainerStarted","Data":"3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a"} Nov 03 07:59:46.492770 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:46.492714 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:46.510420 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:46.510375 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-799454fcf4-24ntl" podStartSLOduration=1.833290181 podStartE2EDuration="5.510361539s" totalImagesPullingTime="3.677071358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:46.508673991 +0000 UTC m=+325.890124910" watchObservedRunningTime="2026-11-03 07:59:46.510361539 +0000 UTC m=+325.891812455" Nov 03 07:59:46.531447 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:46.531392 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" podStartSLOduration=1.91696707 podStartE2EDuration="5.531375832s" totalImagesPullingTime="3.614408762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:46.529758468 +0000 UTC m=+325.911209384" watchObservedRunningTime="2026-11-03 07:59:46.531375832 +0000 UTC m=+325.912826750" Nov 03 07:59:47.521978 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.521939 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77"] Nov 03 07:59:47.525619 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.525588 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.534900 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.534874 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 07:59:47.535041 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.534940 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 07:59:47.535041 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.534966 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 07:59:47.535169 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.535056 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-8g9gt\"" Nov 03 07:59:47.536201 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.536174 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77"] Nov 03 07:59:47.632383 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.632346 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrns\" (UniqueName: \"kubernetes.io/projected/604ea4c3-1290-49a2-8a0a-135cce9cc58b-kube-api-access-wfrns\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.632618 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.632581 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/604ea4c3-1290-49a2-8a0a-135cce9cc58b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.632792 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.632774 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/604ea4c3-1290-49a2-8a0a-135cce9cc58b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.632915 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.632871 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604ea4c3-1290-49a2-8a0a-135cce9cc58b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.733741 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.733709 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrns\" (UniqueName: \"kubernetes.io/projected/604ea4c3-1290-49a2-8a0a-135cce9cc58b-kube-api-access-wfrns\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.733741 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.733745 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/604ea4c3-1290-49a2-8a0a-135cce9cc58b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.733966 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.733791 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/604ea4c3-1290-49a2-8a0a-135cce9cc58b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.733966 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.733812 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604ea4c3-1290-49a2-8a0a-135cce9cc58b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.734395 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.734379 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604ea4c3-1290-49a2-8a0a-135cce9cc58b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.734722 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.734706 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/604ea4c3-1290-49a2-8a0a-135cce9cc58b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.736360 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.736344 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/604ea4c3-1290-49a2-8a0a-135cce9cc58b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.740796 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.740777 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrns\" (UniqueName: \"kubernetes.io/projected/604ea4c3-1290-49a2-8a0a-135cce9cc58b-kube-api-access-wfrns\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rxd77\" (UID: \"604ea4c3-1290-49a2-8a0a-135cce9cc58b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:47.841007 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:47.840968 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 07:59:48.111024 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:48.110955 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77"] Nov 03 07:59:48.114044 ip-10-0-143-47 kubenswrapper[5951]: W1103 07:59:48.114017 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604ea4c3_1290_49a2_8a0a_135cce9cc58b.slice/crio-b2ed35677994e317478d041e2d9a94eb56b491acc48699364f78df3045ec740a WatchSource:0}: Error finding container b2ed35677994e317478d041e2d9a94eb56b491acc48699364f78df3045ec740a: Status 404 returned error can't find the container with id b2ed35677994e317478d041e2d9a94eb56b491acc48699364f78df3045ec740a Nov 03 07:59:48.504741 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:48.504653 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" event={"ID":"604ea4c3-1290-49a2-8a0a-135cce9cc58b","Type":"ContainerStarted","Data":"b2ed35677994e317478d041e2d9a94eb56b491acc48699364f78df3045ec740a"} Nov 03 07:59:57.500618 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.500575 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:57.501122 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.500698 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 07:59:57.554453 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.554419 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ctmbl"] Nov 03 07:59:57.560555 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.560516 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" podUID="85f89838-58cc-491f-9224-21503b685ffd" containerName="manager" containerID="cri-o://3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a" gracePeriod=30 Nov 03 07:59:57.827273 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.827249 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:57.925955 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.925922 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs\") pod \"85f89838-58cc-491f-9224-21503b685ffd\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " Nov 03 07:59:57.926123 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.925974 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d\" (UniqueName: \"kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d\") pod \"85f89838-58cc-491f-9224-21503b685ffd\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " Nov 03 07:59:57.926123 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.926046 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert\") pod \"85f89838-58cc-491f-9224-21503b685ffd\" (UID: \"85f89838-58cc-491f-9224-21503b685ffd\") " Nov 03 07:59:57.928891 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.928845 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs" pod "85f89838-58cc-491f-9224-21503b685ffd" (UID: "85f89838-58cc-491f-9224-21503b685ffd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:59:57.929163 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.929138 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert" pod "85f89838-58cc-491f-9224-21503b685ffd" (UID: "85f89838-58cc-491f-9224-21503b685ffd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:59:57.929281 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:57.929189 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d" pod "85f89838-58cc-491f-9224-21503b685ffd" (UID: "85f89838-58cc-491f-9224-21503b685ffd"). InnerVolumeSpecName "kube-api-access-8th2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:59:58.027413 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.027383 5951 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-webhook-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:59:58.027413 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.027416 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f89838-58cc-491f-9224-21503b685ffd-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:59:58.027627 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.027429 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8th2d\" (UniqueName: \"kubernetes.io/projected/85f89838-58cc-491f-9224-21503b685ffd-kube-api-access-8th2d\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 07:59:58.568334 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.568299 5951 generic.go:356] "Generic (PLEG): container finished" podID="85f89838-58cc-491f-9224-21503b685ffd" containerID="3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a" exitCode=0 Nov 03 07:59:58.568848 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.568361 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" Nov 03 07:59:58.568848 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.568392 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" event={"ID":"85f89838-58cc-491f-9224-21503b685ffd","Type":"ContainerDied","Data":"3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a"} Nov 03 07:59:58.568848 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.568441 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ctmbl" event={"ID":"85f89838-58cc-491f-9224-21503b685ffd","Type":"ContainerDied","Data":"57c1aeb9897e96fbcbe11ba33096ffae6a9ad0268da9dfda7593dc7a251d74d7"} Nov 03 07:59:58.568848 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.568461 5951 scope.go:122] "RemoveContainer" containerID="3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a" Nov 03 07:59:58.578740 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.578721 5951 scope.go:122] "RemoveContainer" containerID="3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a" Nov 03 07:59:58.579024 ip-10-0-143-47 kubenswrapper[5951]: E1103 07:59:58.579004 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a\": container with ID starting with 3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a not found: ID does not exist" containerID="3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a" Nov 03 07:59:58.579099 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.579035 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a"} err="failed to get container status \"3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a\": rpc error: code = NotFound desc = could not find container \"3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a\": container with ID starting with 3e9bfa6eb086db46f1d5e8cd25de8e8b5cd7d9d5bba1e144d3057ff7a376a48a not found: ID does not exist" Nov 03 07:59:58.589478 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.589455 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ctmbl"] Nov 03 07:59:58.593363 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.593332 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ctmbl"] Nov 03 07:59:58.736061 ip-10-0-143-47 kubenswrapper[5951]: I1103 07:59:58.736029 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f89838-58cc-491f-9224-21503b685ffd" path="/var/lib/kubelet/pods/85f89838-58cc-491f-9224-21503b685ffd/volumes" Nov 03 08:00:00.583092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:00.582996 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" event={"ID":"604ea4c3-1290-49a2-8a0a-135cce9cc58b","Type":"ContainerStarted","Data":"b95efe3f0cfcfc05da602482d520b67b4718f2efc9832a8ad41292b068460706"} Nov 03 08:00:00.583092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:00.583061 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 08:00:00.601719 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:00.601662 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" podStartSLOduration=1.394820694 podStartE2EDuration="13.601629118s" totalImagesPullingTime="12.206808424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:00.600355963 +0000 UTC m=+339.981806881" watchObservedRunningTime="2026-11-03 08:00:00.601629118 +0000 UTC m=+339.983080036" Nov 03 08:00:01.954854 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:00:01.954807 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:00:11.594039 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:11.594006 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rxd77" Nov 03 08:00:28.748028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.747993 5951 generic.go:356] "Generic (PLEG): container finished" podID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerID="51df9c11cb64730a622f35a7d66eb442e4b0108b7aeceb406972868fa64f15b1" exitCode=0 Nov 03 08:00:28.748522 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.748039 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" event={"ID":"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1","Type":"ContainerDied","Data":"51df9c11cb64730a622f35a7d66eb442e4b0108b7aeceb406972868fa64f15b1"} Nov 03 08:00:28.748522 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.748414 5951 scope.go:122] "RemoveContainer" containerID="51df9c11cb64730a622f35a7d66eb442e4b0108b7aeceb406972868fa64f15b1" Nov 03 08:00:28.749670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.749599 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:00:28.749786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.749701 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:00:28.749786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.749756 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:00:28.926978 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.926940 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw"] Nov 03 08:00:28.927344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.927331 5951 cpu_manager.go:414] "containerMap: removing container" podUID="85f89838-58cc-491f-9224-21503b685ffd" containerName="manager" Nov 03 08:00:28.927393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.927345 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f89838-58cc-491f-9224-21503b685ffd" containerName="manager" Nov 03 08:00:28.927428 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.927416 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f89838-58cc-491f-9224-21503b685ffd" containerName="manager" Nov 03 08:00:28.949704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.949677 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw"] Nov 03 08:00:28.949826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.949773 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:28.952200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.952176 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 08:00:28.952200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.952184 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 08:00:28.983075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.983044 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 08:00:28.983075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.983044 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 08:00:28.983075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:28.983072 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7nz66\"" Nov 03 08:00:29.028488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.028444 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84q68\" (UniqueName: \"kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.028488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.028484 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.028802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.028518 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.028802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.028551 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.129267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.129230 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84q68\" (UniqueName: \"kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.129267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.129281 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.129536 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.129408 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.129536 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.129439 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.130452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.130398 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.132247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.132224 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.132383 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.132332 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.138346 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.138321 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84q68\" (UniqueName: \"kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68\") pod \"maas-api-7765546dfd-6vbkw\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.287011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.286909 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:29.580976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.580953 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw"] Nov 03 08:00:29.583238 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:00:29.583206 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc028351b_059e_4929_98d3_0cce4713439b.slice/crio-ab8ad3e928909d24fa249076c041db3633467dbb5712b235976ef866855d7ba4 WatchSource:0}: Error finding container ab8ad3e928909d24fa249076c041db3633467dbb5712b235976ef866855d7ba4: Status 404 returned error can't find the container with id ab8ad3e928909d24fa249076c041db3633467dbb5712b235976ef866855d7ba4 Nov 03 08:00:29.755725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.755685 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" event={"ID":"dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1","Type":"ContainerStarted","Data":"3c5ecfcf9a70400469198813b3f8ff1ae8cd4cabdae907ca50af079502b936cd"} Nov 03 08:00:29.757007 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:29.756977 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" event={"ID":"c028351b-059e-4929-98d3-0cce4713439b","Type":"ContainerStarted","Data":"ab8ad3e928909d24fa249076c041db3633467dbb5712b235976ef866855d7ba4"} Nov 03 08:00:30.211909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:30.206464 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 08:00:30.218708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:30.213303 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sn42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body= Nov 03 08:00:30.218708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:30.213376 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podUID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" Nov 03 08:00:31.205551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:31.205081 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sn42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body= Nov 03 08:00:31.205551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:31.205145 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podUID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" Nov 03 08:00:31.955210 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:00:31.955168 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:00:32.217598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:32.217085 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sn42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body= Nov 03 08:00:32.217598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:32.217154 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podUID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" Nov 03 08:00:33.217669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:33.215452 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sn42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body= Nov 03 08:00:33.217669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:33.215511 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podUID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" Nov 03 08:00:33.645418 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:33.645382 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 08:00:33.777336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:33.777299 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" event={"ID":"c028351b-059e-4929-98d3-0cce4713439b","Type":"ContainerStarted","Data":"f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55"} Nov 03 08:00:33.777525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:33.777373 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:00:33.820446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:33.820370 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" podStartSLOduration=2.340213024 podStartE2EDuration="5.820349097s" totalImagesPullingTime="3.480136073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:33.819307098 +0000 UTC m=+373.200758016" watchObservedRunningTime="2026-11-03 08:00:33.820349097 +0000 UTC m=+373.201800015" Nov 03 08:00:34.207174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:34.207129 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sn42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body= Nov 03 08:00:34.207495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:34.207464 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podUID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" Nov 03 08:00:35.208692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:35.208637 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 08:00:35.209572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:35.209556 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" Nov 03 08:00:39.787447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:00:39.787418 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:01:01.953528 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:01:01.953430 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:01:22.251949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.251912 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4"] Nov 03 08:01:22.256256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.256229 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.263972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.263938 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 08:01:22.265980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.265954 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4"] Nov 03 08:01:22.266107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.266087 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vcxhj\"" Nov 03 08:01:22.266169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.266120 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 08:01:22.266169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.266131 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 08:01:22.417561 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.417523 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.417561 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.417561 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.417820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.417601 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.417820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.417704 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cd2\" (UniqueName: \"kubernetes.io/projected/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-kube-api-access-l6cd2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.417820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.417747 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.417975 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.417894 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518431 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518499 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518634 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518703 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518725 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cd2\" (UniqueName: \"kubernetes.io/projected/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-kube-api-access-l6cd2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518765 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518983 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518939 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.518983 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.518971 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.519111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.519084 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.521200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.521178 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.521282 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.521261 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.526737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.526714 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cd2\" (UniqueName: \"kubernetes.io/projected/6cf247f2-4a20-4d4d-b55f-5fb8cd90b354-kube-api-access-l6cd2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9rmr4\" (UID: \"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.576786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.576749 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:22.851219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:22.851090 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4"] Nov 03 08:01:22.854235 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:01:22.854196 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf247f2_4a20_4d4d_b55f_5fb8cd90b354.slice/crio-c59efee6f8a3c860bc87d40425e7e5a0730b3cdc4409bfb02bb3c322cc7d3572 WatchSource:0}: Error finding container c59efee6f8a3c860bc87d40425e7e5a0730b3cdc4409bfb02bb3c322cc7d3572: Status 404 returned error can't find the container with id c59efee6f8a3c860bc87d40425e7e5a0730b3cdc4409bfb02bb3c322cc7d3572 Nov 03 08:01:23.022433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:23.022386 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" event={"ID":"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354","Type":"ContainerStarted","Data":"c59efee6f8a3c860bc87d40425e7e5a0730b3cdc4409bfb02bb3c322cc7d3572"} Nov 03 08:01:25.035156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:25.035116 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" event={"ID":"6cf247f2-4a20-4d4d-b55f-5fb8cd90b354","Type":"ContainerStarted","Data":"5a7e32f265a7527c8b650b6ae2d97fed78ae72f185664889ea337e8b11b41d85"} Nov 03 08:01:25.050419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:25.050374 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" podStartSLOduration=1.654633711 podStartE2EDuration="3.05036132s" totalImagesPullingTime="1.395727609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:25.048741901 +0000 UTC m=+424.430192833" watchObservedRunningTime="2026-11-03 08:01:25.05036132 +0000 UTC m=+424.431812236" Nov 03 08:01:25.577750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:25.577714 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:26.716544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.716487 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99"] Nov 03 08:01:26.721409 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.721389 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.728588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.728027 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 08:01:26.753717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.753689 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99"] Nov 03 08:01:26.861807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.861772 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.862001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.861811 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.862001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.861841 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.862001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.861968 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.862135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.862005 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.862135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.862026 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkjl\" (UniqueName: \"kubernetes.io/projected/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-kube-api-access-qfkjl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963107 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963178 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963210 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkjl\" (UniqueName: \"kubernetes.io/projected/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-kube-api-access-qfkjl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963318 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963347 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963382 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.963864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.963841 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.964140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.964115 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.964282 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.964215 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.967298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.967198 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.969275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.969244 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:26.972249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:26.972226 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkjl\" (UniqueName: \"kubernetes.io/projected/9d85e397-5d01-4200-84d6-ccd89ffdb2a1-kube-api-access-qfkjl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99\" (UID: \"9d85e397-5d01-4200-84d6-ccd89ffdb2a1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:27.046722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:27.046682 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:27.325325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:27.325291 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99"] Nov 03 08:01:27.326698 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:01:27.326670 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d85e397_5d01_4200_84d6_ccd89ffdb2a1.slice/crio-381000027a9d5e01132a5e8eef93a6dbf200212811f28dc571ecdfc208e42137 WatchSource:0}: Error finding container 381000027a9d5e01132a5e8eef93a6dbf200212811f28dc571ecdfc208e42137: Status 404 returned error can't find the container with id 381000027a9d5e01132a5e8eef93a6dbf200212811f28dc571ecdfc208e42137 Nov 03 08:01:28.053716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:28.053674 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" event={"ID":"9d85e397-5d01-4200-84d6-ccd89ffdb2a1","Type":"ContainerStarted","Data":"cd6b7c55bce49e7f40bf75a2cb4d06a470a96b4efdc665e245a1c49f96189634"} Nov 03 08:01:28.054134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:28.053739 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" event={"ID":"9d85e397-5d01-4200-84d6-ccd89ffdb2a1","Type":"ContainerStarted","Data":"381000027a9d5e01132a5e8eef93a6dbf200212811f28dc571ecdfc208e42137"} Nov 03 08:01:28.069933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:28.069880 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" podStartSLOduration=1.7524541139999998 podStartE2EDuration="2.069860973s" totalImagesPullingTime="317.406859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:28.069042297 +0000 UTC m=+427.450493213" watchObservedRunningTime="2026-11-03 08:01:28.069860973 +0000 UTC m=+427.451311891" Nov 03 08:01:29.046821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:29.046784 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:31.604577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.604536 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg"] Nov 03 08:01:31.621676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.621160 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.632811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.632261 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg"] Nov 03 08:01:31.635747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.635699 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 08:01:31.706301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.706269 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.706480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.706444 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.706558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.706500 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.706558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.706529 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/797507d8-79e2-49b2-a51c-e72190a2fd67-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.706700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.706555 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxrm\" (UniqueName: \"kubernetes.io/projected/797507d8-79e2-49b2-a51c-e72190a2fd67-kube-api-access-sbxrm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.706700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.706592 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.807976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.807912 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.808226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.808042 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.808226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.808115 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.808226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.808149 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.808226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.808176 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/797507d8-79e2-49b2-a51c-e72190a2fd67-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.808226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.808201 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxrm\" (UniqueName: \"kubernetes.io/projected/797507d8-79e2-49b2-a51c-e72190a2fd67-kube-api-access-sbxrm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.808999 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.808975 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.809282 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.809252 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.809542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.809522 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.812685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.812623 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/797507d8-79e2-49b2-a51c-e72190a2fd67-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.816285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.816258 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxrm\" (UniqueName: \"kubernetes.io/projected/797507d8-79e2-49b2-a51c-e72190a2fd67-kube-api-access-sbxrm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.817672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.817607 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/797507d8-79e2-49b2-a51c-e72190a2fd67-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m2gsg\" (UID: \"797507d8-79e2-49b2-a51c-e72190a2fd67\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:31.954285 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:01:31.954194 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:01:31.966690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:31.966659 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:32.236933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:32.236860 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg"] Nov 03 08:01:32.241150 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:01:32.241118 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797507d8_79e2_49b2_a51c_e72190a2fd67.slice/crio-2cad95d6fb0d79a7b9ed9cb6ba2bde79075019612c8f64da82b406427aaa0690 WatchSource:0}: Error finding container 2cad95d6fb0d79a7b9ed9cb6ba2bde79075019612c8f64da82b406427aaa0690: Status 404 returned error can't find the container with id 2cad95d6fb0d79a7b9ed9cb6ba2bde79075019612c8f64da82b406427aaa0690 Nov 03 08:01:32.577616 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:32.577594 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:32.590143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:32.590121 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:33.080286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:33.080246 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" event={"ID":"797507d8-79e2-49b2-a51c-e72190a2fd67","Type":"ContainerStarted","Data":"4d77500a09d7f576b3c43a14f99c536c3b631e92676788f25bbe278e443c1a94"} Nov 03 08:01:33.080736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:33.080303 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" event={"ID":"797507d8-79e2-49b2-a51c-e72190a2fd67","Type":"ContainerStarted","Data":"2cad95d6fb0d79a7b9ed9cb6ba2bde79075019612c8f64da82b406427aaa0690"} Nov 03 08:01:33.096431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:33.096378 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" podStartSLOduration=1.78058425 podStartE2EDuration="2.096364342s" totalImagesPullingTime="315.780092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:33.094540445 +0000 UTC m=+432.475991364" watchObservedRunningTime="2026-11-03 08:01:33.096364342 +0000 UTC m=+432.477815258" Nov 03 08:01:33.098765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:33.098740 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" Nov 03 08:01:33.967050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:33.967015 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:37.047495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:37.047436 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:37.064965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:37.064936 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:37.123336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:37.123302 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" Nov 03 08:01:41.967131 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:41.967090 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:41.980118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:41.980093 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:42.141384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:42.141350 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" Nov 03 08:01:50.326889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.326850 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q"] Nov 03 08:01:50.333926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.333904 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.337156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.337122 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 08:01:50.337360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.337134 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 08:01:50.337482 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.337406 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-pg2wh\"" Nov 03 08:01:50.339928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.339902 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q"] Nov 03 08:01:50.382014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.381978 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-85z6q\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.382230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.382043 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wrr\" (UniqueName: \"kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr\") pod \"ai-gateway-controller-7cb8f6bb64-85z6q\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.482852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.482816 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-85z6q\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.483043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.482902 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wrr\" (UniqueName: \"kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr\") pod \"ai-gateway-controller-7cb8f6bb64-85z6q\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.485472 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.485443 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-85z6q\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.490424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.490398 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wrr\" (UniqueName: \"kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr\") pod \"ai-gateway-controller-7cb8f6bb64-85z6q\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.652620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.652511 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:50.930519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:50.930482 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q"] Nov 03 08:01:51.180110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.180022 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" event={"ID":"27b2156a-b44b-47af-acc5-3dd1302ca32a","Type":"ContainerStarted","Data":"6dd6805a9d6269a41d51f88f93d20d326ab7b26f987208182d2b85d6c171a2d2"} Nov 03 08:01:51.186986 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.186949 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95"] Nov 03 08:01:51.194269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.194241 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.203674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.203626 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95"] Nov 03 08:01:51.289005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.288966 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289009 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289265 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289114 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289265 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289175 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9901831a-2efb-4fbc-a245-6c02ed4699de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289269 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289304 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289338 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952hc\" (UniqueName: \"kubernetes.io/projected/9901831a-2efb-4fbc-a245-6c02ed4699de-kube-api-access-952hc\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289398 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.289446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.289419 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390031 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390071 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390106 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390138 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9901831a-2efb-4fbc-a245-6c02ed4699de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390163 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390191 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390218 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952hc\" (UniqueName: \"kubernetes.io/projected/9901831a-2efb-4fbc-a245-6c02ed4699de-kube-api-access-952hc\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390275 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390307 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.390624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390549 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.391049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390714 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.391049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390770 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.391049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.390811 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.391161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.391063 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9901831a-2efb-4fbc-a245-6c02ed4699de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.392779 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.392756 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.393112 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.393090 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.397166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.397141 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9901831a-2efb-4fbc-a245-6c02ed4699de-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.397532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.397511 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952hc\" (UniqueName: \"kubernetes.io/projected/9901831a-2efb-4fbc-a245-6c02ed4699de-kube-api-access-952hc\") pod \"maas-default-gateway-openshift-default-76f4876667-djh95\" (UID: \"9901831a-2efb-4fbc-a245-6c02ed4699de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.511939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.511859 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:51.807345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.806813 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95"] Nov 03 08:01:51.830698 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:01:51.829809 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9901831a_2efb_4fbc_a245_6c02ed4699de.slice/crio-100a771b6098330397ee2fe6814c538e4ebba92fc3ccd0d3b161a40a6091d9d8 WatchSource:0}: Error finding container 100a771b6098330397ee2fe6814c538e4ebba92fc3ccd0d3b161a40a6091d9d8: Status 404 returned error can't find the container with id 100a771b6098330397ee2fe6814c538e4ebba92fc3ccd0d3b161a40a6091d9d8 Nov 03 08:01:51.833223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.833186 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:01:51.833354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.833266 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:01:51.833354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:51.833303 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:01:52.190554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:52.190466 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" event={"ID":"9901831a-2efb-4fbc-a245-6c02ed4699de","Type":"ContainerStarted","Data":"45a49ef7bc19ba52b8a16708f2aa10868b2bc09151c8a22b5777df4bb31818c9"} Nov 03 08:01:52.190554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:52.190534 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" event={"ID":"9901831a-2efb-4fbc-a245-6c02ed4699de","Type":"ContainerStarted","Data":"100a771b6098330397ee2fe6814c538e4ebba92fc3ccd0d3b161a40a6091d9d8"} Nov 03 08:01:52.223196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:52.223131 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podStartSLOduration=1.223109837 podStartE2EDuration="1.223109837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:52.221469434 +0000 UTC m=+451.602920357" watchObservedRunningTime="2026-11-03 08:01:52.223109837 +0000 UTC m=+451.604560761" Nov 03 08:01:52.517964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:52.516939 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:01:52.517964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:52.517579 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:52.517964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:52.517630 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:53.513575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:53.513061 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:53.513575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:53.513147 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:54.241479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:54.241439 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" event={"ID":"27b2156a-b44b-47af-acc5-3dd1302ca32a","Type":"ContainerStarted","Data":"2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f"} Nov 03 08:01:54.242002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:54.241758 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:01:54.259157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:54.259101 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" podStartSLOduration=1.6705971229999999 podStartE2EDuration="4.259087155s" totalImagesPullingTime="2.588490032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:54.257101261 +0000 UTC m=+453.638552178" watchObservedRunningTime="2026-11-03 08:01:54.259087155 +0000 UTC m=+453.640538073" Nov 03 08:01:54.513726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:54.513092 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:54.513726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:54.513173 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:55.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.523052 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:55.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.523121 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:55.572097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.572057 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zbl64"] Nov 03 08:01:55.576259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.576241 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.587284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.587258 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j2rsg\"" Nov 03 08:01:55.588575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.588553 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 08:01:55.589976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.589956 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zbl64"] Nov 03 08:01:55.636763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.636733 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-h9bfr"] Nov 03 08:01:55.640464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.640445 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-h9bfr"] Nov 03 08:01:55.640707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.640579 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.647175 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.647141 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfxl\" (UniqueName: \"kubernetes.io/projected/63a234e8-a52f-40ca-89a3-d54337abfd1e-kube-api-access-2lfxl\") pod \"payload-pre-processing-7b98bb678c-zbl64\" (UID: \"63a234e8-a52f-40ca-89a3-d54337abfd1e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.647294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.647189 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a234e8-a52f-40ca-89a3-d54337abfd1e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zbl64\" (UID: \"63a234e8-a52f-40ca-89a3-d54337abfd1e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.647294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.647276 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1705f8f5-bd72-4db8-8907-bcdb41ea65e4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-h9bfr\" (UID: \"1705f8f5-bd72-4db8-8907-bcdb41ea65e4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.647384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.647363 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5kb\" (UniqueName: \"kubernetes.io/projected/1705f8f5-bd72-4db8-8907-bcdb41ea65e4-kube-api-access-dw5kb\") pod \"payload-processing-55fbbbf9dc-h9bfr\" (UID: \"1705f8f5-bd72-4db8-8907-bcdb41ea65e4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.748791 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.748750 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5kb\" (UniqueName: \"kubernetes.io/projected/1705f8f5-bd72-4db8-8907-bcdb41ea65e4-kube-api-access-dw5kb\") pod \"payload-processing-55fbbbf9dc-h9bfr\" (UID: \"1705f8f5-bd72-4db8-8907-bcdb41ea65e4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.749005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.748979 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfxl\" (UniqueName: \"kubernetes.io/projected/63a234e8-a52f-40ca-89a3-d54337abfd1e-kube-api-access-2lfxl\") pod \"payload-pre-processing-7b98bb678c-zbl64\" (UID: \"63a234e8-a52f-40ca-89a3-d54337abfd1e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.749092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.749016 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a234e8-a52f-40ca-89a3-d54337abfd1e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zbl64\" (UID: \"63a234e8-a52f-40ca-89a3-d54337abfd1e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.749092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.749051 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1705f8f5-bd72-4db8-8907-bcdb41ea65e4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-h9bfr\" (UID: \"1705f8f5-bd72-4db8-8907-bcdb41ea65e4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.749874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.749854 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a234e8-a52f-40ca-89a3-d54337abfd1e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zbl64\" (UID: \"63a234e8-a52f-40ca-89a3-d54337abfd1e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.749955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.749934 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1705f8f5-bd72-4db8-8907-bcdb41ea65e4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-h9bfr\" (UID: \"1705f8f5-bd72-4db8-8907-bcdb41ea65e4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.756448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.756417 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5kb\" (UniqueName: \"kubernetes.io/projected/1705f8f5-bd72-4db8-8907-bcdb41ea65e4-kube-api-access-dw5kb\") pod \"payload-processing-55fbbbf9dc-h9bfr\" (UID: \"1705f8f5-bd72-4db8-8907-bcdb41ea65e4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:55.756601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.756423 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfxl\" (UniqueName: \"kubernetes.io/projected/63a234e8-a52f-40ca-89a3-d54337abfd1e-kube-api-access-2lfxl\") pod \"payload-pre-processing-7b98bb678c-zbl64\" (UID: \"63a234e8-a52f-40ca-89a3-d54337abfd1e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.890939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.890887 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:55.956177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:55.956002 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:01:56.226364 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:01:56.226275 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a234e8_a52f_40ca_89a3_d54337abfd1e.slice/crio-308262adad1119d7f1b6f188b89e96c81dec534ece4aba6eca039ab981010bf0 WatchSource:0}: Error finding container 308262adad1119d7f1b6f188b89e96c81dec534ece4aba6eca039ab981010bf0: Status 404 returned error can't find the container with id 308262adad1119d7f1b6f188b89e96c81dec534ece4aba6eca039ab981010bf0 Nov 03 08:01:56.229158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:56.228881 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zbl64"] Nov 03 08:01:56.255811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:56.255781 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" event={"ID":"63a234e8-a52f-40ca-89a3-d54337abfd1e","Type":"ContainerStarted","Data":"308262adad1119d7f1b6f188b89e96c81dec534ece4aba6eca039ab981010bf0"} Nov 03 08:01:56.288337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:56.288307 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-h9bfr"] Nov 03 08:01:56.516555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:56.516451 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:56.516887 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:56.516828 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:57.264426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:57.264394 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" event={"ID":"1705f8f5-bd72-4db8-8907-bcdb41ea65e4","Type":"ContainerStarted","Data":"14f14110cbd03c1d17cfb6c6ba51f1bda08ee3ad11f6dd5188657be2ea97e2d8"} Nov 03 08:01:57.513433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:57.512990 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:57.513433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:57.513051 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:58.513158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:58.513117 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:58.513582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:58.513192 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:01:59.279523 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:59.279487 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" event={"ID":"63a234e8-a52f-40ca-89a3-d54337abfd1e","Type":"ContainerStarted","Data":"66a4297888ec4a898a021c55da397351f2c8cdacb3eb34b8cf654e3d7418a836"} Nov 03 08:01:59.279718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:59.279606 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:01:59.294863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:59.294767 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" podStartSLOduration=1.475286397 podStartE2EDuration="4.294749592s" totalImagesPullingTime="2.819463195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:59.293331426 +0000 UTC m=+458.674782342" watchObservedRunningTime="2026-11-03 08:01:59.294749592 +0000 UTC m=+458.676200512" Nov 03 08:01:59.512451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:59.512363 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:01:59.512451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:01:59.512415 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:02:01.292977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:01.292940 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" event={"ID":"1705f8f5-bd72-4db8-8907-bcdb41ea65e4","Type":"ContainerStarted","Data":"e37486cfec52eec92e36e183b1436bb3ee6764129406eab5179128e387addb74"} Nov 03 08:02:01.293400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:01.293075 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:02:01.307175 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:01.307086 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" podStartSLOduration=1.7292512279999999 podStartE2EDuration="6.307073514s" totalImagesPullingTime="4.577822286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:01.306360855 +0000 UTC m=+460.687811773" watchObservedRunningTime="2026-11-03 08:02:01.307073514 +0000 UTC m=+460.688524431" Nov 03 08:02:01.553936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:01.553828 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:02:01.554104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:01.553926 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:02:01.962453 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:02:01.962366 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:02:02.514947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:02.514917 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:02:02.515449 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:02.515328 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:02:02.516315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:02.516290 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" Nov 03 08:02:05.249594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.249563 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:02:05.502482 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.502402 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv"] Nov 03 08:02:05.506189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.506167 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.516380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.516352 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv"] Nov 03 08:02:05.556363 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.556325 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a7c571-1963-46f8-abd6-57b3cef729e6-metrics-certs\") pod \"ai-gateway-controller-6c56f47d8c-lv8cv\" (UID: \"13a7c571-1963-46f8-abd6-57b3cef729e6\") " pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.556363 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.556369 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5s5q\" (UniqueName: \"kubernetes.io/projected/13a7c571-1963-46f8-abd6-57b3cef729e6-kube-api-access-s5s5q\") pod \"ai-gateway-controller-6c56f47d8c-lv8cv\" (UID: \"13a7c571-1963-46f8-abd6-57b3cef729e6\") " pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.657695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.657631 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a7c571-1963-46f8-abd6-57b3cef729e6-metrics-certs\") pod \"ai-gateway-controller-6c56f47d8c-lv8cv\" (UID: \"13a7c571-1963-46f8-abd6-57b3cef729e6\") " pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.657695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.657699 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5s5q\" (UniqueName: \"kubernetes.io/projected/13a7c571-1963-46f8-abd6-57b3cef729e6-kube-api-access-s5s5q\") pod \"ai-gateway-controller-6c56f47d8c-lv8cv\" (UID: \"13a7c571-1963-46f8-abd6-57b3cef729e6\") " pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.660223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.660198 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a7c571-1963-46f8-abd6-57b3cef729e6-metrics-certs\") pod \"ai-gateway-controller-6c56f47d8c-lv8cv\" (UID: \"13a7c571-1963-46f8-abd6-57b3cef729e6\") " pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.669064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.669040 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5s5q\" (UniqueName: \"kubernetes.io/projected/13a7c571-1963-46f8-abd6-57b3cef729e6-kube-api-access-s5s5q\") pod \"ai-gateway-controller-6c56f47d8c-lv8cv\" (UID: \"13a7c571-1963-46f8-abd6-57b3cef729e6\") " pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:05.822945 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:05.822907 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:06.100161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.100087 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv"] Nov 03 08:02:06.103423 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:06.103391 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a7c571_1963_46f8_abd6_57b3cef729e6.slice/crio-dc5eab40f4507711f8d7cf63cd348d66a243d68aa3bc3eb80ecabac74ea3c9d0 WatchSource:0}: Error finding container dc5eab40f4507711f8d7cf63cd348d66a243d68aa3bc3eb80ecabac74ea3c9d0: Status 404 returned error can't find the container with id dc5eab40f4507711f8d7cf63cd348d66a243d68aa3bc3eb80ecabac74ea3c9d0 Nov 03 08:02:06.207683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.206375 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6"] Nov 03 08:02:06.211453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.211432 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.224217 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.224185 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6"] Nov 03 08:02:06.264424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264392 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264441 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264465 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264568 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264599 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264631 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264719 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9qnb\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264786 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.264840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.264814 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.324072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.324033 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" event={"ID":"13a7c571-1963-46f8-abd6-57b3cef729e6","Type":"ContainerStarted","Data":"dc5eab40f4507711f8d7cf63cd348d66a243d68aa3bc3eb80ecabac74ea3c9d0"} Nov 03 08:02:06.365928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.365831 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.365928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.365892 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.365928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.365916 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.365932 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qnb\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.365961 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.365981 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366055 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366113 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366149 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366515 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366429 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366515 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366477 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366598 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366616 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.366901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.366884 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.368612 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.368589 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.368705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.368687 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.373088 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.373065 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.373236 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.373218 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9qnb\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb\") pod \"maas-default-gateway-openshift-default-76f4876667-726m6\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.530971 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.530932 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:06.807997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.807969 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6"] Nov 03 08:02:06.809430 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:06.809398 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74eefb2e_d52d_42f9_b10b_10f7efa5c038.slice/crio-0685879cf0823bc82021212eb856988ac66de9f8d66ebb6195d1013848a2b4bf WatchSource:0}: Error finding container 0685879cf0823bc82021212eb856988ac66de9f8d66ebb6195d1013848a2b4bf: Status 404 returned error can't find the container with id 0685879cf0823bc82021212eb856988ac66de9f8d66ebb6195d1013848a2b4bf Nov 03 08:02:06.811824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.811790 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:06.811932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.811876 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:06.811932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:06.811922 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:07.331427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.331392 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" event={"ID":"13a7c571-1963-46f8-abd6-57b3cef729e6","Type":"ContainerStarted","Data":"0d4b3a65ee5bf6a6b8c4f6985e382a5f033c54f2c8426d9f52a53919f8786e76"} Nov 03 08:02:07.331933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.331533 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:07.333104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.333079 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" event={"ID":"74eefb2e-d52d-42f9-b10b-10f7efa5c038","Type":"ContainerStarted","Data":"75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510"} Nov 03 08:02:07.333197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.333123 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" event={"ID":"74eefb2e-d52d-42f9-b10b-10f7efa5c038","Type":"ContainerStarted","Data":"0685879cf0823bc82021212eb856988ac66de9f8d66ebb6195d1013848a2b4bf"} Nov 03 08:02:07.347128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.347083 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" podStartSLOduration=1.867867533 podStartE2EDuration="2.347070433s" totalImagesPullingTime="479.2029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:07.345293199 +0000 UTC m=+466.726744115" watchObservedRunningTime="2026-11-03 08:02:07.347070433 +0000 UTC m=+466.728521350" Nov 03 08:02:07.361946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.361905 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podStartSLOduration=1.361889484 podStartE2EDuration="1.361889484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:07.360439545 +0000 UTC m=+466.741890474" watchObservedRunningTime="2026-11-03 08:02:07.361889484 +0000 UTC m=+466.743340401" Nov 03 08:02:07.531174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.531138 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:07.532612 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.532582 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:07.532730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:07.532677 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:08.532221 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:08.532181 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:08.532675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:08.532238 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:09.531826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:09.531788 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:09.532001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:09.531854 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:10.288115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:10.288087 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zbl64" Nov 03 08:02:10.532245 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:10.532207 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:10.532413 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:10.532276 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:11.532304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:11.532261 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:11.532727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:11.532325 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:12.319596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:12.319566 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-h9bfr" Nov 03 08:02:12.532074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:12.532037 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:12.532274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:12.532102 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:13.531892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:13.531854 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:02:13.532387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:13.531938 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:02:15.085739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:15.085703 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:15.086227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:15.086155 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:15.087354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:15.087335 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:02:18.340787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.340757 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6c56f47d8c-lv8cv" Nov 03 08:02:18.383077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.383040 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q"] Nov 03 08:02:18.383291 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.383262 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" podUID="27b2156a-b44b-47af-acc5-3dd1302ca32a" containerName="manager" containerID="cri-o://2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f" gracePeriod=30 Nov 03 08:02:18.629374 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.629352 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:02:18.718417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.718380 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr\" (UniqueName: \"kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr\") pod \"27b2156a-b44b-47af-acc5-3dd1302ca32a\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " Nov 03 08:02:18.718417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.718420 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs\") pod \"27b2156a-b44b-47af-acc5-3dd1302ca32a\" (UID: \"27b2156a-b44b-47af-acc5-3dd1302ca32a\") " Nov 03 08:02:18.720621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.720579 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs" pod "27b2156a-b44b-47af-acc5-3dd1302ca32a" (UID: "27b2156a-b44b-47af-acc5-3dd1302ca32a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:02:18.720786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.720767 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr" pod "27b2156a-b44b-47af-acc5-3dd1302ca32a" (UID: "27b2156a-b44b-47af-acc5-3dd1302ca32a"). InnerVolumeSpecName "kube-api-access-k7wrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:02:18.819880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.819838 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7wrr\" (UniqueName: \"kubernetes.io/projected/27b2156a-b44b-47af-acc5-3dd1302ca32a-kube-api-access-k7wrr\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:02:18.819880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:18.819876 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b2156a-b44b-47af-acc5-3dd1302ca32a-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:02:19.403584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.403552 5951 generic.go:356] "Generic (PLEG): container finished" podID="27b2156a-b44b-47af-acc5-3dd1302ca32a" containerID="2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f" exitCode=0 Nov 03 08:02:19.404023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.403608 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" Nov 03 08:02:19.404023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.403660 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" event={"ID":"27b2156a-b44b-47af-acc5-3dd1302ca32a","Type":"ContainerDied","Data":"2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f"} Nov 03 08:02:19.404023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.403721 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q" event={"ID":"27b2156a-b44b-47af-acc5-3dd1302ca32a","Type":"ContainerDied","Data":"6dd6805a9d6269a41d51f88f93d20d326ab7b26f987208182d2b85d6c171a2d2"} Nov 03 08:02:19.404023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.403749 5951 scope.go:122] "RemoveContainer" containerID="2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f" Nov 03 08:02:19.413836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.413816 5951 scope.go:122] "RemoveContainer" containerID="2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f" Nov 03 08:02:19.414134 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:02:19.414114 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f\": container with ID starting with 2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f not found: ID does not exist" containerID="2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f" Nov 03 08:02:19.414186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.414145 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f"} err="failed to get container status \"2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f\": rpc error: code = NotFound desc = could not find container \"2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f\": container with ID starting with 2d0913401bad9cdbcddcf158cf6d4098f7c96dca457071e53c68dede6b77261f not found: ID does not exist" Nov 03 08:02:19.418984 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.418960 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q"] Nov 03 08:02:19.422851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:19.422830 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-85z6q"] Nov 03 08:02:20.735542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:20.735513 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b2156a-b44b-47af-acc5-3dd1302ca32a" path="/var/lib/kubelet/pods/27b2156a-b44b-47af-acc5-3dd1302ca32a/volumes" Nov 03 08:02:31.955535 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:02:31.955490 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:02:45.099854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.099816 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk"] Nov 03 08:02:45.101251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.101213 5951 cpu_manager.go:414] "containerMap: removing container" podUID="27b2156a-b44b-47af-acc5-3dd1302ca32a" containerName="manager" Nov 03 08:02:45.101251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.101244 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b2156a-b44b-47af-acc5-3dd1302ca32a" containerName="manager" Nov 03 08:02:45.101425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.101399 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b2156a-b44b-47af-acc5-3dd1302ca32a" containerName="manager" Nov 03 08:02:45.105107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.105083 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.112809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.112782 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-76ef13-openshift-default-dockercfg-4ftn9\"" Nov 03 08:02:45.119554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.119525 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk"] Nov 03 08:02:45.176365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.176335 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.176563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.176374 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.176563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.176472 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.176563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.176512 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.176837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.176569 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.176837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.176657 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.177660 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.177612 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssnv\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.177780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.177684 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.177780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.177743 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.275819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.275784 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr"] Nov 03 08:02:45.279218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279181 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279248 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssnv\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279276 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279325 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279461 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279392 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279461 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279420 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279469 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279498 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.279604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.279533 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.280060 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.280035 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.280489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.280119 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.280489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.280270 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.280489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.280342 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.280912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.280541 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.280912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.280735 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.283523 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.283497 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.283722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.283701 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.289786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.289719 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr"] Nov 03 08:02:45.298003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.297965 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.298147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.298125 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssnv\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.304367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.304338 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e88b7-openshift-default-dockercfg-k6k9p\"" Nov 03 08:02:45.352035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.351956 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c"] Nov 03 08:02:45.373964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.372789 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.376439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.376409 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6bc553-openshift-default-dockercfg-fcnbk\"" Nov 03 08:02:45.380508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.380479 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c"] Nov 03 08:02:45.381045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.380946 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381002 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381101 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57xrt\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381164 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381420 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381688 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381793 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.381935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381864 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.382043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.381999 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.425518 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.425481 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:45.483495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483224 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.483495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483292 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.483495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483437 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483493 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483534 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483558 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483593 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57xrt\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483621 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483682 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483722 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483752 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483824 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.483878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483872 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483911 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nff\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483941 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483966 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.483997 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.484036 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.484095 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.484362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.484316 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.485324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.485298 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.485579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.485557 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.486427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.486259 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.487247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.487227 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.488742 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.488713 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.499024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.497351 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.499024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.498076 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57xrt\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.585082 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585046 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585082 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585089 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585110 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585180 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nff\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585205 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585235 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585270 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585297 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585322 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.585820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.585616 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.586218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.586189 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.587960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.587930 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.588192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.588170 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.588482 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.588460 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.589279 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.589253 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.590631 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.590611 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.600032 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.599998 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.604485 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.604420 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nff\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.612912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.612875 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls"] Nov 03 08:02:45.618868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.618840 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.625948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.625904 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ffe7a8-openshift-default-dockercfg-cp99l\"" Nov 03 08:02:45.630164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.630137 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:45.632047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.632019 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls"] Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686451 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686503 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4z5\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686605 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686632 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686702 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686742 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686809 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686833 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.687055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.686888 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.691855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.691551 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:45.782712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.782046 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk"] Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.788707 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4z5\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.788991 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789020 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789075 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789115 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789159 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789184 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789241 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.796768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.789301 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.797340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.797133 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.797471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.797410 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.798416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.797996 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.798416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.798377 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.798689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.798617 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.803677 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:45.803602 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2606b03_3902_4ea7_9eed_0dd6c35176e6.slice/crio-52e125b04fd0a90577fa5d62eeef3b70c51fff6c6baeba03688bc13b0934d97d WatchSource:0}: Error finding container 52e125b04fd0a90577fa5d62eeef3b70c51fff6c6baeba03688bc13b0934d97d: Status 404 returned error can't find the container with id 52e125b04fd0a90577fa5d62eeef3b70c51fff6c6baeba03688bc13b0934d97d Nov 03 08:02:45.810931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.807052 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:45.810931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.807130 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.810931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.807147 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:45.810931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.807189 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:45.810931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.810859 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.815532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.815431 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.832046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.831977 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4z5\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.934332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.934298 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:45.936454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.936421 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr"] Nov 03 08:02:45.940030 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:45.940003 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b76e67a_2410_43d6_b272_2498ac21edc2.slice/crio-4a7870f3a993a3a83e2c61ee83f4b2b09062bbaf1a950250eddeaee05fdf2170 WatchSource:0}: Error finding container 4a7870f3a993a3a83e2c61ee83f4b2b09062bbaf1a950250eddeaee05fdf2170: Status 404 returned error can't find the container with id 4a7870f3a993a3a83e2c61ee83f4b2b09062bbaf1a950250eddeaee05fdf2170 Nov 03 08:02:45.950340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.950297 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:45.950489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.950393 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:45.950489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:45.950423 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.034969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.034579 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c"] Nov 03 08:02:46.045482 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:46.045436 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497a28d8_1ebd_4c9e_ab47_5f1df765d997.slice/crio-dfd1eb85c4879459258b7c2252a5b266a58f45d947ef1d0c1c869bc3c7a2a1aa WatchSource:0}: Error finding container dfd1eb85c4879459258b7c2252a5b266a58f45d947ef1d0c1c869bc3c7a2a1aa: Status 404 returned error can't find the container with id dfd1eb85c4879459258b7c2252a5b266a58f45d947ef1d0c1c869bc3c7a2a1aa Nov 03 08:02:46.056555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.056488 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.056700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.056603 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.056740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.056693 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.253363 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.252966 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls"] Nov 03 08:02:46.258242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.258208 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.258348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.258272 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.258348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.258300 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:46.550569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.550529 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" event={"ID":"497a28d8-1ebd-4c9e-ab47-5f1df765d997","Type":"ContainerStarted","Data":"d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6"} Nov 03 08:02:46.550828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.550596 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" event={"ID":"497a28d8-1ebd-4c9e-ab47-5f1df765d997","Type":"ContainerStarted","Data":"dfd1eb85c4879459258b7c2252a5b266a58f45d947ef1d0c1c869bc3c7a2a1aa"} Nov 03 08:02:46.552653 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.552620 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" event={"ID":"f2606b03-3902-4ea7-9eed-0dd6c35176e6","Type":"ContainerStarted","Data":"8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a"} Nov 03 08:02:46.552782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.552689 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" event={"ID":"f2606b03-3902-4ea7-9eed-0dd6c35176e6","Type":"ContainerStarted","Data":"52e125b04fd0a90577fa5d62eeef3b70c51fff6c6baeba03688bc13b0934d97d"} Nov 03 08:02:46.554430 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.554350 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" event={"ID":"5b76e67a-2410-43d6-b272-2498ac21edc2","Type":"ContainerStarted","Data":"55d7bb448c21d6e5f91180baafef91ed747fe273818d293b0ccb9f97f7bc1f00"} Nov 03 08:02:46.554430 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.554399 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" event={"ID":"5b76e67a-2410-43d6-b272-2498ac21edc2","Type":"ContainerStarted","Data":"4a7870f3a993a3a83e2c61ee83f4b2b09062bbaf1a950250eddeaee05fdf2170"} Nov 03 08:02:46.556219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.556198 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" event={"ID":"86ebd2f2-96da-4da4-868b-e519288de041","Type":"ContainerStarted","Data":"8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33"} Nov 03 08:02:46.556307 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.556237 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" event={"ID":"86ebd2f2-96da-4da4-868b-e519288de041","Type":"ContainerStarted","Data":"c2d5fec4d3451842e46386366fefc777ab146fda42390a73d88b0bb4700aff42"} Nov 03 08:02:46.571425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.571358 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podStartSLOduration=1.5713434099999999 podStartE2EDuration="1.57134341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:46.568770476 +0000 UTC m=+505.950221393" watchObservedRunningTime="2026-11-03 08:02:46.57134341 +0000 UTC m=+505.952794409" Nov 03 08:02:46.587969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.587906 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podStartSLOduration=1.587876768 podStartE2EDuration="1.587876768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:46.587340923 +0000 UTC m=+505.968791840" watchObservedRunningTime="2026-11-03 08:02:46.587876768 +0000 UTC m=+505.969327686" Nov 03 08:02:46.607459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.607407 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podStartSLOduration=1.607391588 podStartE2EDuration="1.607391588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:46.603748538 +0000 UTC m=+505.985199456" watchObservedRunningTime="2026-11-03 08:02:46.607391588 +0000 UTC m=+505.988842504" Nov 03 08:02:46.625109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.625056 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podStartSLOduration=1.625039398 podStartE2EDuration="1.625039398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:46.622391355 +0000 UTC m=+506.003842272" watchObservedRunningTime="2026-11-03 08:02:46.625039398 +0000 UTC m=+506.006490390" Nov 03 08:02:46.631046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.631016 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:46.692700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.692634 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:46.935043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:46.934945 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:47.425717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.425681 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:47.431733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.431704 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:47.563005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.562960 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:47.564093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.564074 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:02:47.636077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.636048 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:47.699488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.699397 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:47.940773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:47.940745 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:48.568216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:48.568178 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:48.568987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:48.568961 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:48.569111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:48.569008 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:48.569299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:48.569274 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:02:48.569424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:48.569348 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:02:48.569876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:48.569859 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:02:49.973064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.973023 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb"] Nov 03 08:02:49.977892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.977792 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6"] Nov 03 08:02:49.978014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.977893 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:49.981831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.981813 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:49.982765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.982745 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-bd085d-dockercfg-7xms6\"" Nov 03 08:02:49.990262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.990235 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb"] Nov 03 08:02:49.991893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.991867 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-bd085d\"" Nov 03 08:02:49.993230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:49.993206 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6"] Nov 03 08:02:50.045969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.045935 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snl47\" (UniqueName: \"kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47\") pod \"payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.046172 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.046081 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.046217 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.046177 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjrw\" (UniqueName: \"kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw\") pod \"payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.046249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.046217 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.149667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.147538 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snl47\" (UniqueName: \"kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47\") pod \"payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.149667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.147682 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.149667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.147746 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjrw\" (UniqueName: \"kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw\") pod \"payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.149667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.147773 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.149667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.149007 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.150200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.150171 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.160484 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.160453 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjrw\" (UniqueName: \"kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw\") pod \"payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.160607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.160457 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snl47\" (UniqueName: \"kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47\") pod \"payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.303288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.303249 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:50.306321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.306291 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:50.590128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.590098 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb"] Nov 03 08:02:50.591602 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:50.591522 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec249df_0e55_474e_ad96_e8a0769fd2bd.slice/crio-f70a8efcf0078985ca45d21bb27ce6d8b353e278dd19f3f361043497398b87de WatchSource:0}: Error finding container f70a8efcf0078985ca45d21bb27ce6d8b353e278dd19f3f361043497398b87de: Status 404 returned error can't find the container with id f70a8efcf0078985ca45d21bb27ce6d8b353e278dd19f3f361043497398b87de Nov 03 08:02:50.594853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.594827 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6"] Nov 03 08:02:50.598941 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:50.598731 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ecfdea_2575_4a74_aeed_2a9068ae0d0f.slice/crio-b6f40241061fa87d09c137cf234aac3e39fe0eb22009e3518a55c2a27e4a9165 WatchSource:0}: Error finding container b6f40241061fa87d09c137cf234aac3e39fe0eb22009e3518a55c2a27e4a9165: Status 404 returned error can't find the container with id b6f40241061fa87d09c137cf234aac3e39fe0eb22009e3518a55c2a27e4a9165 Nov 03 08:02:50.787230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.787193 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99"] Nov 03 08:02:50.792370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.792339 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:50.801481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.801444 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99"] Nov 03 08:02:50.802882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.802846 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-6bc553\"" Nov 03 08:02:50.957495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.957385 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzv9f\" (UniqueName: \"kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f\") pod \"payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:50.957495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:50.957428 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.059260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.059209 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzv9f\" (UniqueName: \"kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f\") pod \"payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.059260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.059264 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.060164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.060122 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.066603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.066574 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzv9f\" (UniqueName: \"kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f\") pod \"payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.111151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.111119 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.214116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.214032 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj"] Nov 03 08:02:51.219099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.219078 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.229794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.229754 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj"] Nov 03 08:02:51.234539 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.234510 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3e88b7-dockercfg-279bh\"" Nov 03 08:02:51.297815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.297784 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3e88b7\"" Nov 03 08:02:51.365077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.364777 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.365077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.364853 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzw6j\" (UniqueName: \"kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j\") pod \"payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.411611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.411568 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99"] Nov 03 08:02:51.420055 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:51.420015 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddf2738_ce25_43f7_bae6_fc3d3a3c016c.slice/crio-248fcf78a0a647fb036b758f7308333af1a441c37dfba15e61fbf48cd83d64eb WatchSource:0}: Error finding container 248fcf78a0a647fb036b758f7308333af1a441c37dfba15e61fbf48cd83d64eb: Status 404 returned error can't find the container with id 248fcf78a0a647fb036b758f7308333af1a441c37dfba15e61fbf48cd83d64eb Nov 03 08:02:51.466134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.466046 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.466134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.466115 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzw6j\" (UniqueName: \"kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j\") pod \"payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.466816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.466796 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.477055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.477030 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzw6j\" (UniqueName: \"kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j\") pod \"payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.535962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.535927 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:51.590619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.590361 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" event={"ID":"addf2738-ce25-43f7-bae6-fc3d3a3c016c","Type":"ContainerStarted","Data":"8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665"} Nov 03 08:02:51.590619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.590412 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" event={"ID":"addf2738-ce25-43f7-bae6-fc3d3a3c016c","Type":"ContainerStarted","Data":"248fcf78a0a647fb036b758f7308333af1a441c37dfba15e61fbf48cd83d64eb"} Nov 03 08:02:51.590619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.590529 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:02:51.592230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.592200 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" event={"ID":"61ecfdea-2575-4a74-aeed-2a9068ae0d0f","Type":"ContainerStarted","Data":"8ba2726ae3299ec24374c8489192106b95b305a9e21f39d2201a368fa17aef01"} Nov 03 08:02:51.592429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.592256 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" event={"ID":"61ecfdea-2575-4a74-aeed-2a9068ae0d0f","Type":"ContainerStarted","Data":"b6f40241061fa87d09c137cf234aac3e39fe0eb22009e3518a55c2a27e4a9165"} Nov 03 08:02:51.592993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.592970 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:02:51.595499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.595471 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" event={"ID":"aec249df-0e55-474e-ad96-e8a0769fd2bd","Type":"ContainerStarted","Data":"89d7a7844e2224149fca5e7a8a6473ae21b50bceabf7f15751641a830112566d"} Nov 03 08:02:51.595609 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.595523 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" event={"ID":"aec249df-0e55-474e-ad96-e8a0769fd2bd","Type":"ContainerStarted","Data":"f70a8efcf0078985ca45d21bb27ce6d8b353e278dd19f3f361043497398b87de"} Nov 03 08:02:51.595609 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.595570 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:02:51.606025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.605960 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" podStartSLOduration=1.605923006 podStartE2EDuration="1.605923006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:51.604186029 +0000 UTC m=+510.985636945" watchObservedRunningTime="2026-11-03 08:02:51.605923006 +0000 UTC m=+510.987373924" Nov 03 08:02:51.620995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.620939 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" podStartSLOduration=2.620927761 podStartE2EDuration="2.620927761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:51.618217043 +0000 UTC m=+510.999667961" watchObservedRunningTime="2026-11-03 08:02:51.620927761 +0000 UTC m=+511.002378677" Nov 03 08:02:51.638668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.638239 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" podStartSLOduration=2.638217359 podStartE2EDuration="2.638217359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:51.631531301 +0000 UTC m=+511.012982214" watchObservedRunningTime="2026-11-03 08:02:51.638217359 +0000 UTC m=+511.019668277" Nov 03 08:02:51.867009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:51.866973 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj"] Nov 03 08:02:51.874942 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:51.874906 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75476f61_7722_4160_b920_18d413090624.slice/crio-034ae27e0ee14ef69cfba1eb0df6b266527adc5ccb20ffb2c9e764a6944dc3d9 WatchSource:0}: Error finding container 034ae27e0ee14ef69cfba1eb0df6b266527adc5ccb20ffb2c9e764a6944dc3d9: Status 404 returned error can't find the container with id 034ae27e0ee14ef69cfba1eb0df6b266527adc5ccb20ffb2c9e764a6944dc3d9 Nov 03 08:02:52.031964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.031931 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb"] Nov 03 08:02:52.036036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.036010 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.041853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.041787 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ffe7a8\"" Nov 03 08:02:52.041853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.041813 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ffe7a8-dockercfg-qd5z5\"" Nov 03 08:02:52.045528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.045501 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb"] Nov 03 08:02:52.182374 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.182274 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.182850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.182418 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtqw\" (UniqueName: \"kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw\") pod \"payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.283744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.283709 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.283899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.283859 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtqw\" (UniqueName: \"kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw\") pod \"payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.284575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.284552 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.292013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.291983 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtqw\" (UniqueName: \"kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw\") pod \"payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.351518 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.351475 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:52.615708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.612536 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" event={"ID":"75476f61-7722-4160-b920-18d413090624","Type":"ContainerStarted","Data":"2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645"} Nov 03 08:02:52.615708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.612587 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" event={"ID":"75476f61-7722-4160-b920-18d413090624","Type":"ContainerStarted","Data":"034ae27e0ee14ef69cfba1eb0df6b266527adc5ccb20ffb2c9e764a6944dc3d9"} Nov 03 08:02:52.615708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.613416 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:02:52.632691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.632600 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" podStartSLOduration=1.632584396 podStartE2EDuration="1.632584396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:52.629046881 +0000 UTC m=+512.010497800" watchObservedRunningTime="2026-11-03 08:02:52.632584396 +0000 UTC m=+512.014035538" Nov 03 08:02:52.716476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.716353 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb"] Nov 03 08:02:52.724985 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:52.724953 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fecf4d_a101_4feb_9565_e7cb48fb80fe.slice/crio-fb4b585dda97aff8ebfcd831a01be5b872127fb4489e21917dfd1253ce32791b WatchSource:0}: Error finding container fb4b585dda97aff8ebfcd831a01be5b872127fb4489e21917dfd1253ce32791b: Status 404 returned error can't find the container with id fb4b585dda97aff8ebfcd831a01be5b872127fb4489e21917dfd1253ce32791b Nov 03 08:02:52.817108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.817066 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l"] Nov 03 08:02:52.821717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.821687 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:52.826346 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.826313 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-76ef13\"" Nov 03 08:02:52.826517 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.826496 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-76ef13-dockercfg-sghvt\"" Nov 03 08:02:52.829164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.829129 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l"] Nov 03 08:02:52.956226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.956122 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw"] Nov 03 08:02:52.961188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.961160 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:52.968688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.968657 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw"] Nov 03 08:02:52.970185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:52.969490 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-ffe7a8-serving-cert\"" Nov 03 08:02:53.007677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.007596 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.007881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.007853 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dj4\" (UniqueName: \"kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4\") pod \"payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.109124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.109087 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.109314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.109148 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.109414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.109342 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dj4\" (UniqueName: \"kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4\") pod \"payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.109414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.109385 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.109524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.109507 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xz8g\" (UniqueName: \"kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.109597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.109582 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.110479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.110455 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.117485 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.117458 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dj4\" (UniqueName: \"kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4\") pod \"payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.142094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.142027 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.212976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.212127 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xz8g\" (UniqueName: \"kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.212976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.212233 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.212976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.212284 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.212976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.212401 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.214966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.214488 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.216871 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.216842 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.219588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.219547 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.223211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.223180 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xz8g\" (UniqueName: \"kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g\") pod \"maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.281553 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.281520 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:53.463489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.463457 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l"] Nov 03 08:02:53.464032 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:53.464003 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87dd5f9_9463_42d1_89eb_91c0c287521a.slice/crio-efbede0afeb96a67b1b5121645fd195f36be06a80b029514620cadc5f23c9237 WatchSource:0}: Error finding container efbede0afeb96a67b1b5121645fd195f36be06a80b029514620cadc5f23c9237: Status 404 returned error can't find the container with id efbede0afeb96a67b1b5121645fd195f36be06a80b029514620cadc5f23c9237 Nov 03 08:02:53.581290 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.581260 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw"] Nov 03 08:02:53.582160 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:53.582123 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7529824_f718_4ca9_97d8_a0f4529bff1a.slice/crio-848ea544fc02fe77f71e671de91db7019cc953917a0b1b12b8ae49199ebd801f WatchSource:0}: Error finding container 848ea544fc02fe77f71e671de91db7019cc953917a0b1b12b8ae49199ebd801f: Status 404 returned error can't find the container with id 848ea544fc02fe77f71e671de91db7019cc953917a0b1b12b8ae49199ebd801f Nov 03 08:02:53.583694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.583674 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:02:53.633931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.623135 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" event={"ID":"f7529824-f718-4ca9-97d8-a0f4529bff1a","Type":"ContainerStarted","Data":"848ea544fc02fe77f71e671de91db7019cc953917a0b1b12b8ae49199ebd801f"} Nov 03 08:02:53.633931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.629722 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" event={"ID":"b87dd5f9-9463-42d1-89eb-91c0c287521a","Type":"ContainerStarted","Data":"400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423"} Nov 03 08:02:53.633931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.629845 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" event={"ID":"b87dd5f9-9463-42d1-89eb-91c0c287521a","Type":"ContainerStarted","Data":"efbede0afeb96a67b1b5121645fd195f36be06a80b029514620cadc5f23c9237"} Nov 03 08:02:53.633931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.630108 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:02:53.634841 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.634812 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" event={"ID":"46fecf4d-a101-4feb-9565-e7cb48fb80fe","Type":"ContainerStarted","Data":"e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b"} Nov 03 08:02:53.635455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.635431 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" event={"ID":"46fecf4d-a101-4feb-9565-e7cb48fb80fe","Type":"ContainerStarted","Data":"fb4b585dda97aff8ebfcd831a01be5b872127fb4489e21917dfd1253ce32791b"} Nov 03 08:02:53.635593 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.635580 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:02:53.651847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.651585 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" podStartSLOduration=1.6515660319999999 podStartE2EDuration="1.651566032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:53.649142004 +0000 UTC m=+513.030592957" watchObservedRunningTime="2026-11-03 08:02:53.651566032 +0000 UTC m=+513.033016942" Nov 03 08:02:53.666031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.665728 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" podStartSLOduration=1.665708422 podStartE2EDuration="1.665708422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:53.664841946 +0000 UTC m=+513.046292863" watchObservedRunningTime="2026-11-03 08:02:53.665708422 +0000 UTC m=+513.047159696" Nov 03 08:02:53.951317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.951281 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8"] Nov 03 08:02:53.956328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.956305 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:53.962598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.962564 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-19bb0a\"" Nov 03 08:02:53.963627 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.963598 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8"] Nov 03 08:02:53.996781 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:53.996748 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-19bb0a-dockercfg-64t7v\"" Nov 03 08:02:54.123134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.123101 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.123280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.123203 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wk5\" (UniqueName: \"kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5\") pod \"payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.224479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.224383 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.224980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.224493 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wk5\" (UniqueName: \"kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5\") pod \"payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.225409 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.225384 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.232140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.232112 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wk5\" (UniqueName: \"kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5\") pod \"payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.275255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.275221 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:54.556410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.556374 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8"] Nov 03 08:02:54.556627 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:54.556584 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80ff06f_6f59_4664_9b6c_34a6272bf8eb.slice/crio-dcc5021df4d798c923e231721ce9ae807536f9ff1604d6bc01c902ad37befe74 WatchSource:0}: Error finding container dcc5021df4d798c923e231721ce9ae807536f9ff1604d6bc01c902ad37befe74: Status 404 returned error can't find the container with id dcc5021df4d798c923e231721ce9ae807536f9ff1604d6bc01c902ad37befe74 Nov 03 08:02:54.669967 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.669907 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" event={"ID":"c80ff06f-6f59-4664-9b6c-34a6272bf8eb","Type":"ContainerStarted","Data":"dcc5021df4d798c923e231721ce9ae807536f9ff1604d6bc01c902ad37befe74"} Nov 03 08:02:54.672095 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.672060 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" event={"ID":"f7529824-f718-4ca9-97d8-a0f4529bff1a","Type":"ContainerStarted","Data":"009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8"} Nov 03 08:02:54.672691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.672632 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:02:54.691568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:54.691511 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" podStartSLOduration=2.15571975 podStartE2EDuration="2.691491957s" totalImagesPullingTime="535.772207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:54.689113103 +0000 UTC m=+514.070564041" watchObservedRunningTime="2026-11-03 08:02:54.691491957 +0000 UTC m=+514.072942873" Nov 03 08:02:55.682143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:55.682110 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" event={"ID":"c80ff06f-6f59-4664-9b6c-34a6272bf8eb","Type":"ContainerStarted","Data":"b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70"} Nov 03 08:02:55.682517 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:55.682196 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:02:55.703780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:55.702178 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" podStartSLOduration=2.702159017 podStartE2EDuration="2.702159017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:55.700033168 +0000 UTC m=+515.081484096" watchObservedRunningTime="2026-11-03 08:02:55.702159017 +0000 UTC m=+515.083609936" Nov 03 08:02:57.262074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.262043 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2"] Nov 03 08:02:57.267076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.267056 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.274348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.274327 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-bd085d-serving-cert\"" Nov 03 08:02:57.279247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.279222 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2"] Nov 03 08:02:57.378196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.378159 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.378415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.378316 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mfv\" (UniqueName: \"kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.378490 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.378452 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.378553 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.378538 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.480768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.480730 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.481069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.481048 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.481242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.481224 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.481438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.481421 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mfv\" (UniqueName: \"kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.482295 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.482271 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.485321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.485101 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.485321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.485285 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.490528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.490492 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mfv\" (UniqueName: \"kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv\") pod \"maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.591027 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.590794 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:57.984415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:57.984342 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2"] Nov 03 08:02:57.985992 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:02:57.985964 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1598327_6f2f_47c8_963e_4587fbf725cf.slice/crio-51989cfcdbbda552de91becabc7b196c7cf68054447217d4c564dec1390cb9f7 WatchSource:0}: Error finding container 51989cfcdbbda552de91becabc7b196c7cf68054447217d4c564dec1390cb9f7: Status 404 returned error can't find the container with id 51989cfcdbbda552de91becabc7b196c7cf68054447217d4c564dec1390cb9f7 Nov 03 08:02:58.762517 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:58.762439 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" event={"ID":"a1598327-6f2f-47c8-963e-4587fbf725cf","Type":"ContainerStarted","Data":"9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0"} Nov 03 08:02:58.762517 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:58.762488 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" event={"ID":"a1598327-6f2f-47c8-963e-4587fbf725cf","Type":"ContainerStarted","Data":"51989cfcdbbda552de91becabc7b196c7cf68054447217d4c564dec1390cb9f7"} Nov 03 08:02:58.762517 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:58.762509 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:02:58.774988 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:58.774889 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" podStartSLOduration=1.337572953 podStartE2EDuration="1.774873023s" totalImagesPullingTime="437.30007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:58.773709389 +0000 UTC m=+518.155160310" watchObservedRunningTime="2026-11-03 08:02:58.774873023 +0000 UTC m=+518.156323940" Nov 03 08:02:59.364372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.364310 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq"] Nov 03 08:02:59.368896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.368870 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.374008 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.373984 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-15a633-serving-cert\"" Nov 03 08:02:59.376150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.376121 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq"] Nov 03 08:02:59.456685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.447305 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.457118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.457084 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.457296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.457277 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.457556 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.457533 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6gt\" (UniqueName: \"kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.567668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.567520 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.567668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.567603 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.567904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.567705 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6gt\" (UniqueName: \"kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.567904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.567829 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.582732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.569611 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.582732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.578303 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6gt\" (UniqueName: \"kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.582732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.582700 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.583337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.583312 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls\") pod \"maas-api-e2e-worker-w4-15a633-7d64544846-tjczq\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:02:59.685195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:02:59.685102 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:03:00.268300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:00.267974 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq"] Nov 03 08:03:00.363072 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:03:00.345756 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca589f5_8f4e_48b6_8642_44d214a72d28.slice/crio-94edf48b9c7c1cb10eb5424404ec451a8b466498af34f5b386afc7784443a97a WatchSource:0}: Error finding container 94edf48b9c7c1cb10eb5424404ec451a8b466498af34f5b386afc7784443a97a: Status 404 returned error can't find the container with id 94edf48b9c7c1cb10eb5424404ec451a8b466498af34f5b386afc7784443a97a Nov 03 08:03:00.790495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:00.790257 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" event={"ID":"0ca589f5-8f4e-48b6-8642-44d214a72d28","Type":"ContainerStarted","Data":"94edf48b9c7c1cb10eb5424404ec451a8b466498af34f5b386afc7784443a97a"} Nov 03 08:03:01.742411 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:01.740833 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:03:01.857135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:01.857033 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" event={"ID":"0ca589f5-8f4e-48b6-8642-44d214a72d28","Type":"ContainerStarted","Data":"58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb"} Nov 03 08:03:01.857492 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:01.857440 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:03:01.878091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:01.878038 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" podStartSLOduration=2.001206499 podStartE2EDuration="2.878012907s" totalImagesPullingTime="876.806408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:01.87732596 +0000 UTC m=+521.258776878" watchObservedRunningTime="2026-11-03 08:03:01.878012907 +0000 UTC m=+521.259463862" Nov 03 08:03:02.038342 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:03:02.038183 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:03:02.658694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:02.652834 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:03:02.663861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:02.663831 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:03:03.654671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:03.651848 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:03:04.625668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.624773 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:03:04.625668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.624850 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:03:04.625668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.625239 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:03:04.625668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.625281 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:03:04.684846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.684809 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:03:04.685300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.685246 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:03:04.686083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.686037 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:03:04.792675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:04.788973 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:03:07.741537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:07.741504 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:03:08.891548 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:08.891519 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:03:12.602925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:12.025514 5951 generic.go:356] "Generic (PLEG): container finished" podID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerID="55d7bb448c21d6e5f91180baafef91ed747fe273818d293b0ccb9f97f7bc1f00" exitCode=0 Nov 03 08:03:12.602925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:12.025601 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" event={"ID":"5b76e67a-2410-43d6-b272-2498ac21edc2","Type":"ContainerDied","Data":"55d7bb448c21d6e5f91180baafef91ed747fe273818d293b0ccb9f97f7bc1f00"} Nov 03 08:03:12.602925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:12.026100 5951 scope.go:122] "RemoveContainer" containerID="55d7bb448c21d6e5f91180baafef91ed747fe273818d293b0ccb9f97f7bc1f00" Nov 03 08:03:12.602925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:12.027781 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:12.602925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:12.027872 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:12.602925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:12.027917 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.046723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.046679 5951 generic.go:356] "Generic (PLEG): container finished" podID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerID="d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6" exitCode=0 Nov 03 08:03:13.047160 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.047130 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" event={"ID":"497a28d8-1ebd-4c9e-ab47-5f1df765d997","Type":"ContainerDied","Data":"d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6"} Nov 03 08:03:13.047901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.047877 5951 scope.go:122] "RemoveContainer" containerID="d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6" Nov 03 08:03:13.053679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.051228 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.053679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.051317 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.053679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.051360 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.054970 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.054945 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" event={"ID":"5b76e67a-2410-43d6-b272-2498ac21edc2","Type":"ContainerStarted","Data":"56b02f9cf76de38df299646285c62cfb68c3953b0735da8374a10ad55fc28086"} Nov 03 08:03:13.058892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.058805 5951 generic.go:356] "Generic (PLEG): container finished" podID="86ebd2f2-96da-4da4-868b-e519288de041" containerID="8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33" exitCode=0 Nov 03 08:03:13.059015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.058890 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" event={"ID":"86ebd2f2-96da-4da4-868b-e519288de041","Type":"ContainerDied","Data":"8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33"} Nov 03 08:03:13.059699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.059678 5951 scope.go:122] "RemoveContainer" containerID="8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33" Nov 03 08:03:13.063404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.063358 5951 generic.go:356] "Generic (PLEG): container finished" podID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerID="8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a" exitCode=0 Nov 03 08:03:13.063522 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.063410 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" event={"ID":"f2606b03-3902-4ea7-9eed-0dd6c35176e6","Type":"ContainerDied","Data":"8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a"} Nov 03 08:03:13.063953 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.063935 5951 scope.go:122] "RemoveContainer" containerID="8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a" Nov 03 08:03:13.064127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.064089 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.064217 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.064181 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.064275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.064224 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.065787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.065756 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.065882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.065846 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.065942 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.065916 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:03:13.631668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.631079 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:03:13.631668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.631208 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:13.631668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:13.631256 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:14.173148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.172026 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" event={"ID":"497a28d8-1ebd-4c9e-ab47-5f1df765d997","Type":"ContainerStarted","Data":"c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c"} Nov 03 08:03:14.198060 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.193503 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" event={"ID":"86ebd2f2-96da-4da4-868b-e519288de041","Type":"ContainerStarted","Data":"2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177"} Nov 03 08:03:14.220206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.219425 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" event={"ID":"f2606b03-3902-4ea7-9eed-0dd6c35176e6","Type":"ContainerStarted","Data":"fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5"} Nov 03 08:03:14.430235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.430144 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:03:14.430715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.430689 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:14.430913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.430888 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:14.638669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.636352 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:14.638669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.636414 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:14.693960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.692877 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:03:14.694604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.694570 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:14.694876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.694848 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:14.937157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.935815 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:03:14.937615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.937580 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:14.937851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:14.937813 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:15.431058 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.431022 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:15.431370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.431341 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:15.634188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.634148 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:15.634495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.634446 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:15.697354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.697251 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:15.697997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.697949 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:15.936186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.936143 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:15.936560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:15.936514 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:16.428211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.428175 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:16.428545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.428494 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:16.631071 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.631032 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:16.631344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.631310 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:16.696091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.695994 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:16.696361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.696331 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:16.999728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.998154 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:16.999728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:16.998221 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:17.426846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.426803 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:17.427142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.427111 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:17.563977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.563943 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:03:17.635902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.635868 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:17.636557 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.636153 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:17.695166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.695072 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:17.695440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.695406 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:17.941671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.941067 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:17.941671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:17.941144 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:18.428008 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.427950 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:18.428731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.428697 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:18.571671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.568123 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:03:18.620670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.596298 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:03:18.657672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.652387 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:03:18.657909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.652066 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:18.666881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.666838 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:18.695850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.695757 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:18.696099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.696069 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:18.956721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.956180 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:18.956721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:18.956271 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:19.426180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.426147 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:19.426519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.426462 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:19.644700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.643910 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 08:03:19.644700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.643977 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 08:03:19.699784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.697055 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:19.699784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.697117 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:19.935833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.935797 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:19.936124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:19.936094 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:20.426089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.426051 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 08:03:20.426394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.426361 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 08:03:20.695736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.693955 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:03:20.699854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.698761 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:20.699854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.698809 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:20.699854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.699279 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:03:21.028728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:20.981671 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 08:03:21.077470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:21.077433 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 08:03:21.692622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:21.692575 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 08:03:21.692944 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:21.692914 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 08:03:21.975922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:21.975838 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:03:21.988453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:21.988431 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:03:22.162429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:22.162389 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:03:22.169024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:22.168316 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:03:22.697344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:22.697313 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:03:22.698529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:22.698503 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:03:28.674104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.674064 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g"] Nov 03 08:03:28.680551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.680523 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.686697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.686667 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g"] Nov 03 08:03:28.689063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.689027 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"openshift-service-ca.crt\"" Nov 03 08:03:28.689567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.689540 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"kube-root-ca.crt\"" Nov 03 08:03:28.689803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.689783 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"default-dockercfg-lvtgz\"" Nov 03 08:03:28.689898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.689865 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"premium-w3-3e88b7-kserve-self-signed-certs\"" Nov 03 08:03:28.761845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.761804 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.762025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.761954 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.762025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.761989 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.762113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.762028 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.762183 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.762162 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.762226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.762206 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wg9\" (UniqueName: \"kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863234 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863342 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863368 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863420 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863515 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863562 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wg9\" (UniqueName: \"kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863834 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.863954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.863901 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.864173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.864142 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.865800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.865780 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.866426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.866403 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:28.871432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:28.871406 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wg9\" (UniqueName: \"kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9\") pod \"premium-w3-3e88b7-kserve-76469f77f4-8km7g\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:29.015291 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:29.015201 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:29.304693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:29.304662 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g"] Nov 03 08:03:29.305854 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:03:29.305823 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3da169da_7303_4040_85c6_19d02852faa8.slice/crio-5e1c4b231d9d0bcfd303978e5d31ab6fcab3a5a4435c8f0bf0c80f3b32c85497 WatchSource:0}: Error finding container 5e1c4b231d9d0bcfd303978e5d31ab6fcab3a5a4435c8f0bf0c80f3b32c85497: Status 404 returned error can't find the container with id 5e1c4b231d9d0bcfd303978e5d31ab6fcab3a5a4435c8f0bf0c80f3b32c85497 Nov 03 08:03:29.467349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:29.467313 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" event={"ID":"3da169da-7303-4040-85c6-19d02852faa8","Type":"ContainerStarted","Data":"0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf"} Nov 03 08:03:29.467525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:29.467357 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" event={"ID":"3da169da-7303-4040-85c6-19d02852faa8","Type":"ContainerStarted","Data":"5e1c4b231d9d0bcfd303978e5d31ab6fcab3a5a4435c8f0bf0c80f3b32c85497"} Nov 03 08:03:29.484465 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:29.484405 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podStartSLOduration=1.4843861839999999 podStartE2EDuration="1.484386184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:29.482166923 +0000 UTC m=+548.863617830" watchObservedRunningTime="2026-11-03 08:03:29.484386184 +0000 UTC m=+548.865837101" Nov 03 08:03:30.015798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:30.015741 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:31.959837 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:03:31.957308 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:03:38.427794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.427753 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7"] Nov 03 08:03:38.433422 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.433384 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.440315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.440285 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"openshift-service-ca.crt\"" Nov 03 08:03:38.442411 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.440771 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"kube-root-ca.crt\"" Nov 03 08:03:38.442662 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.440898 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"default-dockercfg-fxzvz\"" Nov 03 08:03:38.442803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.440996 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"premium-w0-76ef13-kserve-self-signed-certs\"" Nov 03 08:03:38.446657 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.446618 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7"] Nov 03 08:03:38.581606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.581565 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7l8\" (UniqueName: \"kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.581852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.581679 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.581852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.581718 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.581970 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.581922 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.582028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.581965 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.582173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.582148 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.683910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.683825 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.683910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.683873 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7l8\" (UniqueName: \"kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.683927 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.683951 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.684011 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.684029 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.684300 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.684356 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.684461 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.684412 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.686553 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.686531 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.686848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.686826 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.691500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.691471 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7l8\" (UniqueName: \"kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8\") pod \"premium-w0-76ef13-kserve-dd498c768-znlf7\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:38.753865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:38.753825 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:39.016311 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.016225 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:39.033966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.033933 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:39.035934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.035898 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7"] Nov 03 08:03:39.037695 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:03:39.037638 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d3588f6_d741_4e54_80dc_1ab1caa689ad.slice/crio-602aaea9a23f7e9b25ecbc87b6d9db26de0d3d63adda80c3bca22bb142f3c032 WatchSource:0}: Error finding container 602aaea9a23f7e9b25ecbc87b6d9db26de0d3d63adda80c3bca22bb142f3c032: Status 404 returned error can't find the container with id 602aaea9a23f7e9b25ecbc87b6d9db26de0d3d63adda80c3bca22bb142f3c032 Nov 03 08:03:39.541027 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.540975 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" event={"ID":"7d3588f6-d741-4e54-80dc-1ab1caa689ad","Type":"ContainerStarted","Data":"386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0"} Nov 03 08:03:39.541628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.541035 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" event={"ID":"7d3588f6-d741-4e54-80dc-1ab1caa689ad","Type":"ContainerStarted","Data":"602aaea9a23f7e9b25ecbc87b6d9db26de0d3d63adda80c3bca22bb142f3c032"} Nov 03 08:03:39.552959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.552925 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:03:39.557883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.557812 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podStartSLOduration=1.5577934390000001 podStartE2EDuration="1.557793439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:39.556706785 +0000 UTC m=+558.938157699" watchObservedRunningTime="2026-11-03 08:03:39.557793439 +0000 UTC m=+558.939244365" Nov 03 08:03:39.754110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:39.754077 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:40.917670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.908260 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq"] Nov 03 08:03:40.928690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.921358 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:40.928690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.923408 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq"] Nov 03 08:03:40.930936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.930900 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"openshift-service-ca.crt\"" Nov 03 08:03:40.931540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.931506 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"kube-root-ca.crt\"" Nov 03 08:03:40.931803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.931775 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"default-dockercfg-s8gbj\"" Nov 03 08:03:40.932823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:40.932723 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"premium-w1-19bb0a-kserve-self-signed-certs\"" Nov 03 08:03:41.023078 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.023040 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.023246 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.023193 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.023246 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.023226 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.023403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.023377 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvp7\" (UniqueName: \"kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.023443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.023413 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.023501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.023476 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.132348 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvp7\" (UniqueName: \"kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.132409 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.132450 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.132594 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.132667 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.132690 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.133190 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.133557 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.141667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.133748 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.142262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.142068 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.153736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.152841 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvp7\" (UniqueName: \"kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.154043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.154012 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs\") pod \"premium-w1-19bb0a-kserve-685f6c7566-sqlqq\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.248197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.248117 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:41.559333 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:41.559298 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq"] Nov 03 08:03:41.564329 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:03:41.564294 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ca0f984_9823_43bd_a10e_6fd21931b505.slice/crio-bb40a3d6d0fd1aeae8c55db122b675315bcf48993e8e8a46bcc2f4fc38a78d07 WatchSource:0}: Error finding container bb40a3d6d0fd1aeae8c55db122b675315bcf48993e8e8a46bcc2f4fc38a78d07: Status 404 returned error can't find the container with id bb40a3d6d0fd1aeae8c55db122b675315bcf48993e8e8a46bcc2f4fc38a78d07 Nov 03 08:03:42.564969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:42.564937 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" event={"ID":"1ca0f984-9823-43bd-a10e-6fd21931b505","Type":"ContainerStarted","Data":"1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94"} Nov 03 08:03:42.565381 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:42.564977 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" event={"ID":"1ca0f984-9823-43bd-a10e-6fd21931b505","Type":"ContainerStarted","Data":"bb40a3d6d0fd1aeae8c55db122b675315bcf48993e8e8a46bcc2f4fc38a78d07"} Nov 03 08:03:42.583026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:42.582968 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podStartSLOduration=2.582952834 podStartE2EDuration="2.582952834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:42.580977799 +0000 UTC m=+561.962428719" watchObservedRunningTime="2026-11-03 08:03:42.582952834 +0000 UTC m=+561.964403752" Nov 03 08:03:43.248956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:43.248922 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:48.754757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:48.754721 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:48.780988 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:48.780952 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:49.654629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:49.654595 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:03:51.248424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:51.248377 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:51.261863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:51.261836 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:03:51.663369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:03:51.663328 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:04:01.956144 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:04:01.956105 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:04:20.832011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:20.831950 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:04:20.835289 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:20.835269 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:04:30.333542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.333505 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9"] Nov 03 08:04:30.339229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.339207 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.346828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.346804 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-bd085d-openshift-default-dockercfg-v65xr\"" Nov 03 08:04:30.348632 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.348610 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9"] Nov 03 08:04:30.431587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.431548 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54l5c\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.431797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.431623 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.431797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.431773 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.431896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.431807 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.431896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.431891 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.432019 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.432001 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.432072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.432054 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.432124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.432088 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.432203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.432178 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.533604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533570 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.533818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533628 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.533818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533671 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.533818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533697 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.533818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533742 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54l5c\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.533818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533793 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533889 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533919 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.533993 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.534387 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.534405 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.534633 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.534881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.534856 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.535150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.535128 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.537062 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.537035 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.537275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.537253 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.542181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.542156 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.543339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.543316 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54l5c\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.599354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.599256 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75"] Nov 03 08:04:30.604625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.604602 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.612126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.612041 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-15a633-openshift-default-dockercfg-vdsjd\"" Nov 03 08:04:30.614105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.614072 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75"] Nov 03 08:04:30.656014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.655981 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:30.743109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743069 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743168 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743341 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743406 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743438 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743467 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrdv\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743526 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743550 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.743608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.743587 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845132 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845093 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845276 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845382 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845360 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845492 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845561 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845588 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845612 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrdv\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845684 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.845715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845713 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.846014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845738 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.846014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845864 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.846014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.845886 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.846184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.846118 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.846453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.846430 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.848174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.848153 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.848370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.848348 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.854835 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.854777 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.854975 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.854954 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrdv\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.926481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.926449 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:30.956685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.955535 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9"] Nov 03 08:04:30.956685 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:04:30.956081 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c83392b_7a26_4cfc_9998_a8a8a5ccf89a.slice/crio-399f3b5814defca8348b07893c0b70ea72679c15af3e03de93434cf580cffe04 WatchSource:0}: Error finding container 399f3b5814defca8348b07893c0b70ea72679c15af3e03de93434cf580cffe04: Status 404 returned error can't find the container with id 399f3b5814defca8348b07893c0b70ea72679c15af3e03de93434cf580cffe04 Nov 03 08:04:30.959141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.959108 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:04:30.959244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.959193 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:04:30.959244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:30.959238 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:04:31.218315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.218277 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75"] Nov 03 08:04:31.219690 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:04:31.219661 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391919b4_ac7b_4656_ab8c_9650666c3d64.slice/crio-2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac WatchSource:0}: Error finding container 2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac: Status 404 returned error can't find the container with id 2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac Nov 03 08:04:31.222785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.222747 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:04:31.222875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.222829 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:04:31.222942 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.222872 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:04:31.960373 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:04:31.960247 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:04:31.965699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.965655 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" event={"ID":"391919b4-ac7b-4656-ab8c-9650666c3d64","Type":"ContainerStarted","Data":"2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc"} Nov 03 08:04:31.965898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.965739 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" event={"ID":"391919b4-ac7b-4656-ab8c-9650666c3d64","Type":"ContainerStarted","Data":"2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac"} Nov 03 08:04:31.967491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.967455 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" event={"ID":"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a","Type":"ContainerStarted","Data":"8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940"} Nov 03 08:04:31.967618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.967509 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" event={"ID":"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a","Type":"ContainerStarted","Data":"399f3b5814defca8348b07893c0b70ea72679c15af3e03de93434cf580cffe04"} Nov 03 08:04:31.989410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:31.984872 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podStartSLOduration=1.984851861 podStartE2EDuration="1.984851861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:31.982270933 +0000 UTC m=+611.363721863" watchObservedRunningTime="2026-11-03 08:04:31.984851861 +0000 UTC m=+611.366302780" Nov 03 08:04:32.006673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.004396 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podStartSLOduration=2.00434548 podStartE2EDuration="2.00434548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:32.002667478 +0000 UTC m=+611.384118412" watchObservedRunningTime="2026-11-03 08:04:32.00434548 +0000 UTC m=+611.385796397" Nov 03 08:04:32.657180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.657137 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:32.658585 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.658558 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 08:04:32.658721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.658613 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 08:04:32.938427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.938326 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:32.938621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.938479 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 08:04:32.938621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:32.938527 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 08:04:33.656670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:33.656377 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 08:04:33.656670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:33.656439 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 08:04:33.934728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:33.934086 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 08:04:33.934728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:33.934159 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 08:04:34.660669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:34.660305 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 08:04:34.660669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:34.660379 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 08:04:34.927730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:34.927159 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 08:04:34.927730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:34.927230 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 08:04:35.659046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:35.659007 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 08:04:35.659345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:35.659316 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 08:04:35.938728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:35.927068 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 08:04:35.938728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:35.927133 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 08:04:36.659665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:36.659107 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 08:04:36.659665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:36.659189 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 08:04:36.934215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:36.934082 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 08:04:36.934215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:36.934169 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 08:04:37.656365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:37.656332 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 08:04:37.656895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:37.656410 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 08:04:37.927385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:37.926952 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 08:04:37.927385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:37.927030 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 08:04:38.661471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:38.661437 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:38.661922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:38.661903 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:38.662879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:38.662857 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:04:38.932197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:38.932110 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:38.932601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:38.932574 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:38.933443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:38.933423 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:04:42.750988 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.750954 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w"] Nov 03 08:04:42.756171 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.756133 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w"] Nov 03 08:04:42.756327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.756276 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.759496 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.759470 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\"" Nov 03 08:04:42.759801 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.759608 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 08:04:42.862432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.862388 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.862910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.862883 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.863184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.863151 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxpg\" (UniqueName: \"kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.964190 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.964149 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.964365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.964228 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxpg\" (UniqueName: \"kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.964365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.964307 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.965141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.965113 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.966797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.966772 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:42.972107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:42.972072 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxpg\" (UniqueName: \"kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg\") pod \"maas-api-revoke-sub-e4771903-kkp9w\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:43.073812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:43.073770 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:43.369746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:43.369714 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w"] Nov 03 08:04:43.371201 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:04:43.371171 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ccdb9_8fda_4ba2_9ac2_bbdb5483f669.slice/crio-6283c2d663d9967ebb020b9dda4df343b5c3831392f33f882ba386b8a0018fbd WatchSource:0}: Error finding container 6283c2d663d9967ebb020b9dda4df343b5c3831392f33f882ba386b8a0018fbd: Status 404 returned error can't find the container with id 6283c2d663d9967ebb020b9dda4df343b5c3831392f33f882ba386b8a0018fbd Nov 03 08:04:44.148908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:44.148868 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" event={"ID":"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669","Type":"ContainerStarted","Data":"6283c2d663d9967ebb020b9dda4df343b5c3831392f33f882ba386b8a0018fbd"} Nov 03 08:04:48.184507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:48.184461 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" event={"ID":"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669","Type":"ContainerStarted","Data":"2c5ea55d528a6a543aa24cfe9061a3f5eb6b00c0a6845cf1ab38be0d599d4a37"} Nov 03 08:04:48.232672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:48.232306 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" podStartSLOduration=2.426395408 podStartE2EDuration="6.232287214s" totalImagesPullingTime="3.805891806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:48.218366679 +0000 UTC m=+627.599817601" watchObservedRunningTime="2026-11-03 08:04:48.232287214 +0000 UTC m=+627.613738139" Nov 03 08:04:49.192599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:49.192555 5951 generic.go:356] "Generic (PLEG): container finished" podID="9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" containerID="2c5ea55d528a6a543aa24cfe9061a3f5eb6b00c0a6845cf1ab38be0d599d4a37" exitCode=0 Nov 03 08:04:49.193037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:49.192663 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" event={"ID":"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669","Type":"ContainerDied","Data":"2c5ea55d528a6a543aa24cfe9061a3f5eb6b00c0a6845cf1ab38be0d599d4a37"} Nov 03 08:04:50.370370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.370342 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:50.508717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.508608 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca\") pod \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " Nov 03 08:04:50.509112 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.508781 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca\") pod \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " Nov 03 08:04:50.509112 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.508920 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg\" (UniqueName: \"kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg\") pod \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\" (UID: \"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669\") " Nov 03 08:04:50.509296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.509232 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca" pod "9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" (UID: "9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:04:50.509445 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.509425 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-openshift-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:04:50.511711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.511675 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg" pod "9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" (UID: "9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669"). InnerVolumeSpecName "kube-api-access-zgxpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:04:50.511834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.511726 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca" pod "9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" (UID: "9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:04:50.610897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.610863 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-maas-api-xks-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:04:50.610897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:50.610893 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxpg\" (UniqueName: \"kubernetes.io/projected/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669-kube-api-access-zgxpg\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:04:51.213385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:51.213341 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" event={"ID":"9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669","Type":"ContainerDied","Data":"6283c2d663d9967ebb020b9dda4df343b5c3831392f33f882ba386b8a0018fbd"} Nov 03 08:04:51.213597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:51.213408 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6283c2d663d9967ebb020b9dda4df343b5c3831392f33f882ba386b8a0018fbd" Nov 03 08:04:51.213597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:51.213420 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w" Nov 03 08:04:54.926656 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.926613 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf"] Nov 03 08:04:54.927505 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.927485 5951 cpu_manager.go:414] "containerMap: removing container" podUID="9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" containerName="revoke-keys" Nov 03 08:04:54.927584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.927508 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" containerName="revoke-keys" Nov 03 08:04:54.927682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.927670 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" containerName="revoke-keys" Nov 03 08:04:54.931923 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.931894 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:54.935694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.934745 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 08:04:54.935694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.935347 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\"" Nov 03 08:04:54.939320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:54.939290 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf"] Nov 03 08:04:55.071630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.071590 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.071863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.071786 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.071943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.071921 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5jr\" (UniqueName: \"kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.172804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.172764 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5jr\" (UniqueName: \"kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.173000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.172852 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.173000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.172932 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.173559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.173528 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.175660 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.175620 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.183681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.183590 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5jr\" (UniqueName: \"kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr\") pod \"maas-api-revoke-sub-3539c6a9-s5pjf\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.255478 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.255217 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:55.547073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:55.547037 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf"] Nov 03 08:04:56.254818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:56.254717 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" event={"ID":"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb","Type":"ContainerStarted","Data":"596f38a4634bc585167824ea1f4ba912d8d60f45d34220fa215a356da337019d"} Nov 03 08:04:56.255530 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:56.255473 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" event={"ID":"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb","Type":"ContainerStarted","Data":"62b6e3d2fefda1bcb913682a25b92c7ee935c0c2cde97450d0b1b3ad9548bf15"} Nov 03 08:04:56.274900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:56.274840 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" podStartSLOduration=2.2748252669999998 podStartE2EDuration="2.274825267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:56.271611413 +0000 UTC m=+635.653062332" watchObservedRunningTime="2026-11-03 08:04:56.274825267 +0000 UTC m=+635.656276232" Nov 03 08:04:57.264776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:57.264734 5951 generic.go:356] "Generic (PLEG): container finished" podID="3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" containerID="596f38a4634bc585167824ea1f4ba912d8d60f45d34220fa215a356da337019d" exitCode=0 Nov 03 08:04:57.265163 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:57.264806 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" event={"ID":"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb","Type":"ContainerDied","Data":"596f38a4634bc585167824ea1f4ba912d8d60f45d34220fa215a356da337019d"} Nov 03 08:04:58.453894 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.453865 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:58.512111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.512061 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca\") pod \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " Nov 03 08:04:58.512373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.512241 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr\" (UniqueName: \"kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr\") pod \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " Nov 03 08:04:58.512557 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.512537 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca\") pod \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\" (UID: \"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb\") " Nov 03 08:04:58.513596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.513570 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca" pod "3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" (UID: "3df9a36c-3ee2-403c-bec6-9ec23c8b5acb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:04:58.515145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.515112 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr" pod "3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" (UID: "3df9a36c-3ee2-403c-bec6-9ec23c8b5acb"). InnerVolumeSpecName "kube-api-access-kb5jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:04:58.516266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.516239 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca" pod "3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" (UID: "3df9a36c-3ee2-403c-bec6-9ec23c8b5acb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:04:58.613879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.613842 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb5jr\" (UniqueName: \"kubernetes.io/projected/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-kube-api-access-kb5jr\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:04:58.613879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.613878 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-xks-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:04:58.613879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:58.613889 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb-maas-api-openshift-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:04:59.279919 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:59.279883 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" Nov 03 08:04:59.279919 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:59.279912 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf" event={"ID":"3df9a36c-3ee2-403c-bec6-9ec23c8b5acb","Type":"ContainerDied","Data":"62b6e3d2fefda1bcb913682a25b92c7ee935c0c2cde97450d0b1b3ad9548bf15"} Nov 03 08:04:59.280153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:04:59.279962 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b6e3d2fefda1bcb913682a25b92c7ee935c0c2cde97450d0b1b3ad9548bf15" Nov 03 08:05:02.258949 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:05:02.258905 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:05:05.323676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.323281 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw"] Nov 03 08:05:05.327973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.324234 5951 cpu_manager.go:414] "containerMap: removing container" podUID="3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" containerName="revoke-keys" Nov 03 08:05:05.327973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.324259 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" containerName="revoke-keys" Nov 03 08:05:05.327973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.324441 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" containerName="revoke-keys" Nov 03 08:05:05.336418 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.335148 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.344054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.341626 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"default-dockercfg-xt4tf\"" Nov 03 08:05:05.344054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.341837 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw"] Nov 03 08:05:05.344054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.342124 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"distinct-w2-6bc553-kserve-self-signed-certs\"" Nov 03 08:05:05.344054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.342184 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"kube-root-ca.crt\"" Nov 03 08:05:05.344054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.342251 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"openshift-service-ca.crt\"" Nov 03 08:05:05.358131 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.358093 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.358305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.358263 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.358354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.358326 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.358392 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.358366 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.358527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.358505 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2z5\" (UniqueName: \"kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.358563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.358553 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.460452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.460383 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.460712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.460543 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2z5\" (UniqueName: \"kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.460712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.460591 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.462492 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.462465 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.463268 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.463243 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.463414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.463395 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.464227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.464203 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.478685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.478633 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.481372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.481345 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.482816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.481613 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.483059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.482689 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.486500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.486452 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2z5\" (UniqueName: \"kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5\") pod \"distinct-w2-6bc553-kserve-84d89655b7-m7jnw\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.655286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.655197 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:05.952497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:05.952469 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw"] Nov 03 08:05:05.954102 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:05.954073 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e89968_3a4b_4703_be77_07bfe931a99b.slice/crio-7ae61db8a742c17596e5889431b0deea9badde237a45251720c2d28cf4c092f8 WatchSource:0}: Error finding container 7ae61db8a742c17596e5889431b0deea9badde237a45251720c2d28cf4c092f8: Status 404 returned error can't find the container with id 7ae61db8a742c17596e5889431b0deea9badde237a45251720c2d28cf4c092f8 Nov 03 08:05:06.433077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:06.433019 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" event={"ID":"d8e89968-3a4b-4703-be77-07bfe931a99b","Type":"ContainerStarted","Data":"2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6"} Nov 03 08:05:06.433077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:06.433083 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" event={"ID":"d8e89968-3a4b-4703-be77-07bfe931a99b","Type":"ContainerStarted","Data":"7ae61db8a742c17596e5889431b0deea9badde237a45251720c2d28cf4c092f8"} Nov 03 08:05:06.450616 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:06.450552 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podStartSLOduration=1.450536449 podStartE2EDuration="1.450536449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:06.447457966 +0000 UTC m=+645.828908883" watchObservedRunningTime="2026-11-03 08:05:06.450536449 +0000 UTC m=+645.831987366" Nov 03 08:05:06.655622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:06.655577 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:11.662664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.662612 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx"] Nov 03 08:05:11.669455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.669430 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.677322 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.677237 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"distinct-w0-76ef13-kserve-self-signed-certs\"" Nov 03 08:05:11.681849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.681817 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx"] Nov 03 08:05:11.748587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.748547 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.748803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.748631 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvn46\" (UniqueName: \"kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.748803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.748762 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.748803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.748797 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.748962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.748944 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.749006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.748974 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.849990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.849953 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.849990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.849996 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850055 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850094 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvn46\" (UniqueName: \"kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850223 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850272 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850410 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850565 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.850688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.850634 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.852574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.852543 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.852736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.852719 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.856980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.856957 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvn46\" (UniqueName: \"kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46\") pod \"distinct-w0-76ef13-kserve-8595ddffc9-v7dsx\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:11.999993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:11.999893 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:12.289006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:12.288980 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx"] Nov 03 08:05:12.290932 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:12.290903 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb310d0c8_f990_466f_b5bf_9a388124ef84.slice/crio-7cb4f0f3411eb7703e0d5684b64dcb5f5b1b9766b0bb7e63b2b4233cf2ec1769 WatchSource:0}: Error finding container 7cb4f0f3411eb7703e0d5684b64dcb5f5b1b9766b0bb7e63b2b4233cf2ec1769: Status 404 returned error can't find the container with id 7cb4f0f3411eb7703e0d5684b64dcb5f5b1b9766b0bb7e63b2b4233cf2ec1769 Nov 03 08:05:12.477012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:12.476966 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" event={"ID":"b310d0c8-f990-466f-b5bf-9a388124ef84","Type":"ContainerStarted","Data":"a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7"} Nov 03 08:05:12.477012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:12.477012 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" event={"ID":"b310d0c8-f990-466f-b5bf-9a388124ef84","Type":"ContainerStarted","Data":"7cb4f0f3411eb7703e0d5684b64dcb5f5b1b9766b0bb7e63b2b4233cf2ec1769"} Nov 03 08:05:12.494982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:12.494920 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podStartSLOduration=1.494903903 podStartE2EDuration="1.494903903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:12.491728163 +0000 UTC m=+651.873179081" watchObservedRunningTime="2026-11-03 08:05:12.494903903 +0000 UTC m=+651.876354819" Nov 03 08:05:13.000306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:13.000251 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:15.666672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:15.662896 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:15.941069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:15.936810 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:16.045763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.045706 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn"] Nov 03 08:05:16.298927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.298880 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn"] Nov 03 08:05:16.299288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.299271 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.319018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.318987 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-19bb0a-openshift-default-dockercfg-2phtd\"" Nov 03 08:05:16.475974 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.475930 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.476299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.476277 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.476434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.476416 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd4s8\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.476545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.476530 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.476732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.476716 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.476901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.476883 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.477166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.477148 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.477326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.477311 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.477468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.477453 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.578616 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.578518 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.578888 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.578861 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd4s8\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.579049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579032 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.579212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579194 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.579347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579331 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.580135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579435 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.580135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579469 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.580135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579515 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.580135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.579592 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.580741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.580503 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.581116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.581094 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.581571 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.581248 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.584434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.584409 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.585126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.585083 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.585611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.585591 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.588460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.588409 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.589898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.589875 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd4s8\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.591808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.591712 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:16.603076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.602770 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:05:16.738153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:16.738120 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:17.048627 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.048565 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn"] Nov 03 08:05:17.048991 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:17.048949 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaab243_c773_42cb_b917_d7ec04455bf1.slice/crio-f53e229e4adf031c1125741862116de07888ab83325a4b79baffd2e28e0988b2 WatchSource:0}: Error finding container f53e229e4adf031c1125741862116de07888ab83325a4b79baffd2e28e0988b2: Status 404 returned error can't find the container with id f53e229e4adf031c1125741862116de07888ab83325a4b79baffd2e28e0988b2 Nov 03 08:05:17.053396 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.053355 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:17.053592 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.053564 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:17.053765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.053622 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:17.612664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.611914 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" event={"ID":"dcaab243-c773-42cb-b917-d7ec04455bf1","Type":"ContainerStarted","Data":"eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf"} Nov 03 08:05:17.612664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.611980 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" event={"ID":"dcaab243-c773-42cb-b917-d7ec04455bf1","Type":"ContainerStarted","Data":"f53e229e4adf031c1125741862116de07888ab83325a4b79baffd2e28e0988b2"} Nov 03 08:05:17.640891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.640598 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podStartSLOduration=2.640567399 podStartE2EDuration="2.640567399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:17.637870869 +0000 UTC m=+657.019321822" watchObservedRunningTime="2026-11-03 08:05:17.640567399 +0000 UTC m=+657.022018317" Nov 03 08:05:17.742721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:17.738568 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:18.740383 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:18.740346 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 08:05:18.740634 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:18.740411 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 08:05:19.739280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:19.739242 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 08:05:19.739714 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:19.739305 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 08:05:20.752997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:20.752958 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 08:05:20.753720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:20.753691 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 08:05:21.738992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:21.738953 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 08:05:21.739181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:21.739018 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 08:05:22.015142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:22.014055 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:22.039607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:22.039575 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:22.677106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:22.677077 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:05:22.741809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:22.741772 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 08:05:22.742079 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:22.742048 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 08:05:23.670677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.670622 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2"] Nov 03 08:05:23.677010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.676974 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.685390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.685352 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"distinct-w4-15a633-kserve-self-signed-certs\"" Nov 03 08:05:23.688227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.687166 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2"] Nov 03 08:05:23.688370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.687346 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"default-dockercfg-mgwxq\"" Nov 03 08:05:23.691407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.691386 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"openshift-service-ca.crt\"" Nov 03 08:05:23.694869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.694843 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"kube-root-ca.crt\"" Nov 03 08:05:23.739133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.739093 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 08:05:23.739318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.739170 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 08:05:23.842670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.842399 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.842670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.842513 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.842670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.842552 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.842670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.842636 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.843061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.842687 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2ks\" (UniqueName: \"kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.843061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.842750 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.947730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.947408 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.947730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.947505 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.947730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.947543 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.947730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.947625 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.947730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.947673 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2ks\" (UniqueName: \"kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.947730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.947735 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.953763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.948719 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.953763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.949440 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.953763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.949550 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.966168 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.966134 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.967917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.967884 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.972185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.972117 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2ks\" (UniqueName: \"kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks\") pod \"distinct-w4-15a633-kserve-58b98467d-rz7r2\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:23.993966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:23.993930 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:24.290928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.290898 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2"] Nov 03 08:05:24.292476 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:24.292449 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5472f604_f001_48f8_a95a_4680d038e507.slice/crio-04c3137f7d06c2464da2494c33184f2c7313e7e70a5b296fc90f07b0104359c6 WatchSource:0}: Error finding container 04c3137f7d06c2464da2494c33184f2c7313e7e70a5b296fc90f07b0104359c6: Status 404 returned error can't find the container with id 04c3137f7d06c2464da2494c33184f2c7313e7e70a5b296fc90f07b0104359c6 Nov 03 08:05:24.682113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.682016 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" event={"ID":"5472f604-f001-48f8-a95a-4680d038e507","Type":"ContainerStarted","Data":"730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52"} Nov 03 08:05:24.682113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.682070 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" event={"ID":"5472f604-f001-48f8-a95a-4680d038e507","Type":"ContainerStarted","Data":"04c3137f7d06c2464da2494c33184f2c7313e7e70a5b296fc90f07b0104359c6"} Nov 03 08:05:24.702465 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.702401 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podStartSLOduration=1.7023845359999998 podStartE2EDuration="1.702384536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:24.699786677 +0000 UTC m=+664.081237595" watchObservedRunningTime="2026-11-03 08:05:24.702384536 +0000 UTC m=+664.083835487" Nov 03 08:05:24.744033 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.744000 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:24.744326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.744306 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:24.745070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.745053 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:05:24.995320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:24.995236 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:30.478698 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.478663 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42"] Nov 03 08:05:30.484149 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.484123 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42"] Nov 03 08:05:30.484293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.484214 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.542834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.542803 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq"] Nov 03 08:05:30.547890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.547865 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.562710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.562675 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq"] Nov 03 08:05:30.578591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578563 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.578776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578600 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.578776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578669 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkvc\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.578935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578771 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.578935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578830 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.578935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578857 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.578935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578884 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.579158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.578929 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.579158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579002 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.579158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579067 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.579158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579112 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.579158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579139 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.579401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579173 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.579401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579242 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.579401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579303 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.579401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579330 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.579659 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579393 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tmz\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.579659 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.579481 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.680213 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680175 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.680397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680229 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.680397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680255 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.680397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680279 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.680397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680307 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.680397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680348 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.680397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680383 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.680747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680407 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.680747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680442 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tmz\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.680747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680491 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.680747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680615 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.680747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680705 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.681002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680766 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkvc\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.681002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680803 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.681002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680869 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.681002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680898 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.681002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680902 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.681002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680950 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.681636 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.680928 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.681793 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.682379 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.682424 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.682735 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.683133 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.683531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.683377 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.684118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.683688 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.684118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.683959 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.686190 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.684340 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.693797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.693767 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.694128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.694101 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.694578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.694520 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.697572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.697545 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tmz\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.698462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.698441 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkvc\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.699383 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.699358 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.699489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.699392 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.805031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.804996 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:30.859815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.859773 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq"] Nov 03 08:05:30.863922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.863889 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:30.865391 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.865367 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.882108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.882080 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq"] Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884018 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884058 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884147 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884175 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884222 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884265 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884313 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnj5\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884391 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.885110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.884418 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986284 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnj5\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986484 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986359 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986484 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986387 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986486 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986510 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986588 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.986788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.986615 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.987124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.987098 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.987266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.987146 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.987339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.987277 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.987339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.987330 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.987458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.987367 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.987748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.987723 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.988319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.988296 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.989947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.989927 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.992096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.992074 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:30.996438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.996413 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:31.000096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:30.999700 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnj5\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:31.110017 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:31.109986 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd997040_85b1_4ebf_be01_04de25803871.slice/crio-f978b80dcb0e1611079ee454647b283e0ca0d3407c483c20b461e49f4b1cac76 WatchSource:0}: Error finding container f978b80dcb0e1611079ee454647b283e0ca0d3407c483c20b461e49f4b1cac76: Status 404 returned error can't find the container with id f978b80dcb0e1611079ee454647b283e0ca0d3407c483c20b461e49f4b1cac76 Nov 03 08:05:31.112483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.112457 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42"] Nov 03 08:05:31.113593 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.113556 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.113709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.113689 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.113756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.113743 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.191425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.191386 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq"] Nov 03 08:05:31.191571 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:31.191547 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1ca7f5_b4a8_4ce0_97e2_091d8b204cd4.slice/crio-537b6f0dc2c3e1394f2056872b84e21e97c2ec747468ff607f712cab6311e2e1 WatchSource:0}: Error finding container 537b6f0dc2c3e1394f2056872b84e21e97c2ec747468ff607f712cab6311e2e1: Status 404 returned error can't find the container with id 537b6f0dc2c3e1394f2056872b84e21e97c2ec747468ff607f712cab6311e2e1 Nov 03 08:05:31.196664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.196606 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.196781 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.196741 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.196840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.196790 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.208679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.208599 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:31.524097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.524062 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq"] Nov 03 08:05:31.528780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.528739 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.528897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.528824 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.528897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.528877 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:31.766377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.766263 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" event={"ID":"fd997040-85b1-4ebf-be01-04de25803871","Type":"ContainerStarted","Data":"7d3c1979df6083c862845b52bf6454ca90f33a025f8ef8ba6e2dc6a5352773ae"} Nov 03 08:05:31.766377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.766338 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" event={"ID":"fd997040-85b1-4ebf-be01-04de25803871","Type":"ContainerStarted","Data":"f978b80dcb0e1611079ee454647b283e0ca0d3407c483c20b461e49f4b1cac76"} Nov 03 08:05:31.768894 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.768859 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" event={"ID":"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4","Type":"ContainerStarted","Data":"a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a"} Nov 03 08:05:31.769047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.768925 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" event={"ID":"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4","Type":"ContainerStarted","Data":"537b6f0dc2c3e1394f2056872b84e21e97c2ec747468ff607f712cab6311e2e1"} Nov 03 08:05:31.771034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.771010 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" event={"ID":"a974d7b3-dd54-4006-b009-736db4c87ece","Type":"ContainerStarted","Data":"44b82842792bc458d877214cc73f3e0af1a24a2906034460e5b4dac6a86dac90"} Nov 03 08:05:31.771157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.771058 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" event={"ID":"a974d7b3-dd54-4006-b009-736db4c87ece","Type":"ContainerStarted","Data":"a77f905f8734ebca175e9d3c07124b15b64274b5530f39a87ba5b11706850bf6"} Nov 03 08:05:31.787263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.787214 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podStartSLOduration=1.787200038 podStartE2EDuration="1.787200038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:31.783247662 +0000 UTC m=+671.164698601" watchObservedRunningTime="2026-11-03 08:05:31.787200038 +0000 UTC m=+671.168650954" Nov 03 08:05:31.806073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.806036 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:31.810840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.810784 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podStartSLOduration=1.810764423 podStartE2EDuration="1.810764423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:31.80351288 +0000 UTC m=+671.184963810" watchObservedRunningTime="2026-11-03 08:05:31.810764423 +0000 UTC m=+671.192215339" Nov 03 08:05:31.851706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.851624 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podStartSLOduration=1.851602951 podStartE2EDuration="1.851602951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:31.845626148 +0000 UTC m=+671.227077065" watchObservedRunningTime="2026-11-03 08:05:31.851602951 +0000 UTC m=+671.233053869" Nov 03 08:05:31.864895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:31.864863 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:31.957151 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:05:31.957111 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:05:32.213917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.211086 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:32.223162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.214798 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:32.223162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.214865 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:32.904155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.894055 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 08:05:32.904155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.894116 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 08:05:32.928973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.928940 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 08:05:32.929230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:32.929203 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 08:05:33.265258 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.265168 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:33.266171 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.266137 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:33.810670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.807196 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 08:05:33.810670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.807254 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 08:05:33.898436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.896597 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 08:05:33.898436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.896681 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 08:05:33.997564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:33.995015 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:34.217087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.209348 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:34.217087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.209416 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:34.388010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.387123 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:34.836801 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.832222 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 08:05:34.836801 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.832358 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 08:05:34.864581 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.864522 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 08:05:34.864931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:34.864882 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 08:05:35.167150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.167055 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:05:35.216667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.212268 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:35.216667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.212340 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:35.826693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.812785 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 08:05:35.826693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.812849 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 08:05:35.865684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.865109 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 08:05:35.865684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:35.865175 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 08:05:36.210845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:36.210754 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:36.211419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:36.211386 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:36.847684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:36.847151 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 08:05:36.847684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:36.847234 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 08:05:36.865158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:36.865120 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 08:05:36.865804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:36.865414 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 08:05:37.214119 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:37.214022 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:37.214703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:37.214669 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:37.817673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:37.808072 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 08:05:37.817673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:37.808133 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 08:05:37.864744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:37.864690 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 08:05:37.865439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:37.865015 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 08:05:38.211161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:38.211055 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 08:05:38.211428 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:38.211395 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 08:05:39.213193 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.208671 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:39.213193 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.211573 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:39.213193 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.212925 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:05:39.299418 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.298628 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:39.300536 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.300388 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:39.318948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.318912 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:05:39.636155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.636118 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:39.636567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.636541 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:39.637533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:39.637514 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:05:45.670595 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.670557 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5"] Nov 03 08:05:45.675109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.675081 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.686512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.685387 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5"] Nov 03 08:05:45.697960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.697933 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45"] Nov 03 08:05:45.705994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.705969 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.722012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.721972 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722104 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722170 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722199 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722224 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722324 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722368 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722482 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.722587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.722568 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6bl\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.723729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.723705 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45"] Nov 03 08:05:45.823699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.823661 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.823935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.823784 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.823935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.823855 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.823935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.823909 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.823966 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.823992 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824014 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824037 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.824314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824104 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.824314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824143 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.824314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824165 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.824314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824205 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824241 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824290 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824328 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.824614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824383 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6bl\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.824614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824415 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxnzk\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.824614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.824481 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.825350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.825322 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.826542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.826516 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.826761 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.826735 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.826886 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.826808 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.827072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.827049 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.828110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.828091 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.828189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.828165 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.832268 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.832248 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.833763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.833746 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6bl\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:45.925798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.925701 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.925798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.925754 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.925815 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.925908 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.925954 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926002 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926031 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926145 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926212 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxnzk\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926229 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926241 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926528 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926545 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.926794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.926702 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.929027 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.928999 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.929191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.929171 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.933731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.933700 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.933907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.933887 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxnzk\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:45.999853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:45.999806 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:46.030545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.030441 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq"] Nov 03 08:05:46.043580 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.043532 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:46.050745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.050710 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq"] Nov 03 08:05:46.050884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.050834 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.058976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.058475 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-76ef13\"/\"distinct-2-w0-76ef13-kserve-self-signed-certs\"" Nov 03 08:05:46.133407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.129920 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.133407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.129970 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.133407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.130007 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.133407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.130191 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.133407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.130214 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.133407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.130305 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbcb\" (UniqueName: \"kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.237284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.237221 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.238812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.237573 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.238812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.237635 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.238812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.237702 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.238812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.237722 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.238812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.237816 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbcb\" (UniqueName: \"kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.239174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.239000 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.240580 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.240541 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.240877 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.240838 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.244552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.244477 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.251862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.251280 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.277740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.275942 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbcb\" (UniqueName: \"kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb\") pod \"distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.319827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.319796 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5"] Nov 03 08:05:46.321118 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:46.321092 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d49e8d_1e4a_4397_b8c4_9169b3fdb4cf.slice/crio-7d7200c62d9248ed9b19a851580f0b309c174d9b8751e56a6b04295718bbecee WatchSource:0}: Error finding container 7d7200c62d9248ed9b19a851580f0b309c174d9b8751e56a6b04295718bbecee: Status 404 returned error can't find the container with id 7d7200c62d9248ed9b19a851580f0b309c174d9b8751e56a6b04295718bbecee Nov 03 08:05:46.324463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.324392 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:46.324555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.324516 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:46.324610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.324562 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:46.360273 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.360245 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45"] Nov 03 08:05:46.361816 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:46.361790 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded23459d_9aa8_492e_872d_5676cab351b4.slice/crio-69b17c5e29cc06a7911d9c8de2a6eaf21a349105c7d1e909a992f63fefc5e118 WatchSource:0}: Error finding container 69b17c5e29cc06a7911d9c8de2a6eaf21a349105c7d1e909a992f63fefc5e118: Status 404 returned error can't find the container with id 69b17c5e29cc06a7911d9c8de2a6eaf21a349105c7d1e909a992f63fefc5e118 Nov 03 08:05:46.364314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.364277 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:46.364406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.364360 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:46.364454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.364401 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:05:46.377227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.377204 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:46.508525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.494445 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" event={"ID":"ed23459d-9aa8-492e-872d-5676cab351b4","Type":"ContainerStarted","Data":"ddfd53388cdf1675e5639ae62a6366e1a37840143924922df23636f3e197139b"} Nov 03 08:05:46.508525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.494548 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" event={"ID":"ed23459d-9aa8-492e-872d-5676cab351b4","Type":"ContainerStarted","Data":"69b17c5e29cc06a7911d9c8de2a6eaf21a349105c7d1e909a992f63fefc5e118"} Nov 03 08:05:46.508525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.499745 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" event={"ID":"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf","Type":"ContainerStarted","Data":"aa9656116cb1d99b3502145e020939c547c95ba18b1315bd815adea2912148ba"} Nov 03 08:05:46.508525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.499799 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" event={"ID":"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf","Type":"ContainerStarted","Data":"7d7200c62d9248ed9b19a851580f0b309c174d9b8751e56a6b04295718bbecee"} Nov 03 08:05:46.582954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.554969 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podStartSLOduration=1.554947179 podStartE2EDuration="1.554947179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:46.543506322 +0000 UTC m=+685.924957239" watchObservedRunningTime="2026-11-03 08:05:46.554947179 +0000 UTC m=+685.936398097" Nov 03 08:05:46.582954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.555329 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podStartSLOduration=1.555319213 podStartE2EDuration="1.555319213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:46.520312948 +0000 UTC m=+685.901763867" watchObservedRunningTime="2026-11-03 08:05:46.555319213 +0000 UTC m=+685.936770131" Nov 03 08:05:46.677706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:46.677677 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq"] Nov 03 08:05:46.679394 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:05:46.679338 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcedc1ead_9210_438f_b524_aebf4b93942e.slice/crio-cf40f93fb3643e1acc8e292d89c1023dd6e16669d643d911eb440c0c44d7ea89 WatchSource:0}: Error finding container cf40f93fb3643e1acc8e292d89c1023dd6e16669d643d911eb440c0c44d7ea89: Status 404 returned error can't find the container with id cf40f93fb3643e1acc8e292d89c1023dd6e16669d643d911eb440c0c44d7ea89 Nov 03 08:05:47.004066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.003981 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:47.006810 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.006774 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:47.007047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.007023 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:47.054443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.054410 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:47.056603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.056510 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:47.056861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.056836 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:47.535787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.535742 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" event={"ID":"cedc1ead-9210-438f-b524-aebf4b93942e","Type":"ContainerStarted","Data":"f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766"} Nov 03 08:05:47.536064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.536045 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" event={"ID":"cedc1ead-9210-438f-b524-aebf4b93942e","Type":"ContainerStarted","Data":"cf40f93fb3643e1acc8e292d89c1023dd6e16669d643d911eb440c0c44d7ea89"} Nov 03 08:05:47.563943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:47.563880 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podStartSLOduration=1.5638560080000001 podStartE2EDuration="1.563856008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:47.562154357 +0000 UTC m=+686.943605276" watchObservedRunningTime="2026-11-03 08:05:47.563856008 +0000 UTC m=+686.945306925" Nov 03 08:05:48.005201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:48.005106 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:48.005845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:48.005813 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:48.044146 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:48.044106 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:48.044407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:48.044381 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:48.378667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:48.377982 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:49.017671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:49.017047 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:49.017671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:49.017107 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:49.052848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:49.052815 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:49.053101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:49.053077 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:50.011669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:50.004097 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:50.011669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:50.004172 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:50.054671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:50.047046 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:50.054671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:50.047105 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:51.006672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:51.002024 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:51.006672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:51.002098 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:51.056670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:51.044034 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:51.056670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:51.044111 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:52.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:52.010733 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:52.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:52.010802 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:52.045669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:52.044995 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:52.045669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:52.045067 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:53.001284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:53.001238 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 08:05:53.001815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:53.001307 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 08:05:53.043992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:53.043947 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 08:05:53.044185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:53.044014 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 08:05:54.008697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:54.007629 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:54.009241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:54.009115 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:54.012551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:54.011573 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:05:54.049561 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:54.049520 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:54.050575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:54.050554 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:54.051439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:54.051421 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:05:56.377939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:56.377898 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:56.391170 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:56.391145 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:56.704547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:56.704451 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:05:59.577754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.572251 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2"] Nov 03 08:05:59.580091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.579565 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.583196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.582296 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2"] Nov 03 08:05:59.644668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.620985 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"distinct-2-w4-15a633-kserve-self-signed-certs\"" Nov 03 08:05:59.663890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.663841 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.664165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.664127 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.667430 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.667402 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.667796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.667773 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.675057 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.675032 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.675472 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.675381 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4q5\" (UniqueName: \"kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.841107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.841032 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.841682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.841664 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.842157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.842140 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.842460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.842447 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4q5\" (UniqueName: \"kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.845798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.842411 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.845798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.841573 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.846357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.846342 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.846479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.846454 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.846560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.846551 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.846852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.846836 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.850748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.850571 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.854165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.854121 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4q5\" (UniqueName: \"kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5\") pod \"distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:05:59.935720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:05:59.934766 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:00.249806 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.249768 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2"] Nov 03 08:06:00.381223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.381188 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99"] Nov 03 08:06:00.386376 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.386348 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.400450 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.400416 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99"] Nov 03 08:06:00.457264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457223 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457474 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457292 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngfr\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457474 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457424 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457495 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457514 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457532 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457551 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457582 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.457602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.457597 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558490 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558458 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558516 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngfr\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558574 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558630 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558669 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558692 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558716 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558764 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.558949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.558788 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.559538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.559112 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.559538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.559217 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.559538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.559311 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.559538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.559341 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.559538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.559487 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.561338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.561318 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.561419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.561403 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.567965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.567937 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.568655 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.568613 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngfr\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.704857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.704825 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:00.822799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.817736 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v"] Nov 03 08:06:00.828722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.826927 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.835860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.834147 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v"] Nov 03 08:06:00.844880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.844125 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" event={"ID":"9a561f60-a65e-4eeb-b7b2-b832818dc2bc","Type":"ContainerStarted","Data":"cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd"} Nov 03 08:06:00.844880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.844170 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" event={"ID":"9a561f60-a65e-4eeb-b7b2-b832818dc2bc","Type":"ContainerStarted","Data":"f334db3ca21b27664edacd40bb8684fe37d47d1be30ab84017d4c2c010e05b2e"} Nov 03 08:06:00.863357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863318 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863446 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863519 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863545 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863626 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863669 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863751 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863822 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c85x\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.863909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.863855 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.871343 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.871265 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podStartSLOduration=1.871246364 podStartE2EDuration="1.871246364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:00.86447065 +0000 UTC m=+700.245921568" watchObservedRunningTime="2026-11-03 08:06:00.871246364 +0000 UTC m=+700.252697282" Nov 03 08:06:00.935623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.935589 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:00.965251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965216 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.965427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965304 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.965427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965343 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.965427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965358 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.965601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965433 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.965601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965476 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965602 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965727 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965742 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965727 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c85x\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965797 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.965938 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.966188 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.966357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.966339 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.967929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.967906 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.968233 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.968217 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.972376 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.972334 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c85x\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:00.972815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:00.972786 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:01.020188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.018931 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99"] Nov 03 08:06:01.042558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.042515 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:01.042713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.042581 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:01.042713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.042609 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:01.151559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.151532 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:01.459176 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:01.459137 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c21cf61_4d1e_41a0_adef_173cce70680a.slice/crio-abf799cc6cfef5eb85afda017f0cb220ee89a7ef4eebb9a8ed9c2f6f676b1704 WatchSource:0}: Error finding container abf799cc6cfef5eb85afda017f0cb220ee89a7ef4eebb9a8ed9c2f6f676b1704: Status 404 returned error can't find the container with id abf799cc6cfef5eb85afda017f0cb220ee89a7ef4eebb9a8ed9c2f6f676b1704 Nov 03 08:06:01.459390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.459368 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v"] Nov 03 08:06:01.461913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.461877 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:01.462008 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.461990 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:01.462061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.462035 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:01.858856 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.858815 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" event={"ID":"aa445227-7a5e-4b11-8fd6-55fdef0cc618","Type":"ContainerStarted","Data":"9ecd7b0ddadff368de7d0854e0b46d7c6b5a846757833ffa3da46bb4667cbc25"} Nov 03 08:06:01.859313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.858873 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" event={"ID":"aa445227-7a5e-4b11-8fd6-55fdef0cc618","Type":"ContainerStarted","Data":"27c4b6579f8762831149c8f136dc5504dc8055cf1204c7fe253da1d9a87589d3"} Nov 03 08:06:01.865185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.865150 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" event={"ID":"3c21cf61-4d1e-41a0-adef-173cce70680a","Type":"ContainerStarted","Data":"61a6301ddd5247d62149a43aa3964451af25105f0cfa415b7210c11b8404739c"} Nov 03 08:06:01.865423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.865397 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" event={"ID":"3c21cf61-4d1e-41a0-adef-173cce70680a","Type":"ContainerStarted","Data":"abf799cc6cfef5eb85afda017f0cb220ee89a7ef4eebb9a8ed9c2f6f676b1704"} Nov 03 08:06:01.882054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:01.881987 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podStartSLOduration=1.881969535 podStartE2EDuration="1.881969535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:01.876908827 +0000 UTC m=+701.258359748" watchObservedRunningTime="2026-11-03 08:06:01.881969535 +0000 UTC m=+701.263420453" Nov 03 08:06:01.970719 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:06:01.970678 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:06:02.151834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:02.151755 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:02.153432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:02.153404 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:02.153545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:02.153461 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:02.714672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:02.711626 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:02.714672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:02.711791 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 08:06:02.714672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:02.711834 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 08:06:03.155134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:03.155086 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:03.155689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:03.155152 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:03.708679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:03.708042 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 08:06:03.708679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:03.708103 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 08:06:04.160673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:04.155891 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:04.160673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:04.155967 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:04.709671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:04.708065 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 08:06:04.709671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:04.708126 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 08:06:05.154669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:05.153288 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:05.154669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:05.153351 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:05.713670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:05.707350 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 08:06:05.713670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:05.707425 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 08:06:06.153669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:06.152724 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:06.153669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:06.152803 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:06.712668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:06.712054 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 08:06:06.712668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:06.712116 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 08:06:07.157669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:07.156748 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:07.157669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:07.156810 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:07.705533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:07.705496 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 08:06:07.705824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:07.705795 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 08:06:08.152960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:08.152920 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 08:06:08.153132 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:08.152993 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 08:06:09.086369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.082745 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:09.086369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.083836 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:09.086369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.085585 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:06:09.102438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.102379 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podStartSLOduration=9.102366369 podStartE2EDuration="9.102366369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:01.915104568 +0000 UTC m=+701.296555496" watchObservedRunningTime="2026-11-03 08:06:09.102366369 +0000 UTC m=+708.483817285" Nov 03 08:06:09.194757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.194725 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:09.195916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.195860 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:09.197926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.197902 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:06:09.935535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.935500 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:09.949423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.949394 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:09.987474 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:09.987442 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:11.621083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.621049 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv"] Nov 03 08:06:11.627300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.627274 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.630833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.630802 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"distinct-2-w1-19bb0a-kserve-self-signed-certs\"" Nov 03 08:06:11.646615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.646587 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv"] Nov 03 08:06:11.652248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.652211 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.652386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.652340 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.652500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.652480 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.652565 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.652522 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.652745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.652705 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxcc\" (UniqueName: \"kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.652745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.652740 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.753930 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.753885 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxcc\" (UniqueName: \"kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.754163 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.753944 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.754163 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.754022 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.754312 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.754164 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.754312 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.754209 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.754312 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.754242 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.755142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.755112 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.755383 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.755358 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.763256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.758262 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.770912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.766247 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.770912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.770701 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.799388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.799354 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxcc\" (UniqueName: \"kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc\") pod \"distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:11.956356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:11.956267 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:12.247811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:12.247786 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv"] Nov 03 08:06:12.249292 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:12.249261 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b5ba96c_79e3_4ed4_8fca_f42eb0d61ca0.slice/crio-6453163236d056b05319b07f38d414af793beef5ea20140f8cebcbb8350ba028 WatchSource:0}: Error finding container 6453163236d056b05319b07f38d414af793beef5ea20140f8cebcbb8350ba028: Status 404 returned error can't find the container with id 6453163236d056b05319b07f38d414af793beef5ea20140f8cebcbb8350ba028 Nov 03 08:06:13.004848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:13.004808 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" event={"ID":"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0","Type":"ContainerStarted","Data":"c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884"} Nov 03 08:06:13.005297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:13.004865 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" event={"ID":"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0","Type":"ContainerStarted","Data":"6453163236d056b05319b07f38d414af793beef5ea20140f8cebcbb8350ba028"} Nov 03 08:06:13.022000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:13.021930 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podStartSLOduration=2.021909106 podStartE2EDuration="2.021909106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:13.019157997 +0000 UTC m=+712.400608915" watchObservedRunningTime="2026-11-03 08:06:13.021909106 +0000 UTC m=+712.403360024" Nov 03 08:06:13.956841 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:13.956797 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:15.503881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.503838 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj"] Nov 03 08:06:15.520668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.518141 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.561493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.559387 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj"] Nov 03 08:06:15.633952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.633898 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4"] Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.643435 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644670 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644698 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644747 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644786 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644804 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644854 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644894 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644970 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.647720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.644989 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdgv\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.657510 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.656231 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4"] Nov 03 08:06:15.746007 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.745969 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746018 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdgv\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746083 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746111 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746186 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746243 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746272 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746307 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jnr\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.746417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746341 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.746417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746411 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.746727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746478 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746625 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.746727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746624 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746724 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.746898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746734 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746763 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746827 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.746898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.746872 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.747197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.747178 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.747288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.747274 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.747352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.747309 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.747352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.747316 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.747352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.747340 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.749347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.749321 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.749602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.749587 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.767287 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.767216 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.767395 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.767285 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdgv\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.848811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.848763 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.848994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.848822 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.848994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.848843 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849063 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jnr\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849225 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849201 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849254 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849202 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849293 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849373 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849446 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849496 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849523 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849751 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.849876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.849857 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.851870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.851853 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.852112 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.852091 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.856546 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.856527 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jnr\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.856702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.856682 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:15.896252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.896219 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:15.983507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:15.974094 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:16.193620 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:16.193589 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff0891c_7d56_4964_b6c0_1e92bd2ac1ef.slice/crio-edcfb11468cbba84e92f63ce0292b54a689c9d22b6c916da757c8a446d691ea1 WatchSource:0}: Error finding container edcfb11468cbba84e92f63ce0292b54a689c9d22b6c916da757c8a446d691ea1: Status 404 returned error can't find the container with id edcfb11468cbba84e92f63ce0292b54a689c9d22b6c916da757c8a446d691ea1 Nov 03 08:06:16.194248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.194220 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj"] Nov 03 08:06:16.197017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.196980 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:16.197088 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.197068 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:16.197133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.197110 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:16.292433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.292398 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4"] Nov 03 08:06:16.292728 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:16.292701 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba08e75a_59aa_49ae_97ed_ca66fe8387c6.slice/crio-e00ffb918e3e07180aa44be55d634190fa6b169de6ced85175cf8366d416f761 WatchSource:0}: Error finding container e00ffb918e3e07180aa44be55d634190fa6b169de6ced85175cf8366d416f761: Status 404 returned error can't find the container with id e00ffb918e3e07180aa44be55d634190fa6b169de6ced85175cf8366d416f761 Nov 03 08:06:16.295595 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.295558 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:16.295713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.295689 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:16.295775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.295744 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:16.326867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.323414 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4"] Nov 03 08:06:16.346807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.346177 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4"] Nov 03 08:06:16.346807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.346308 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.352682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.352550 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e88b7\"/\"distinct-2-w3-3e88b7-kserve-self-signed-certs\"" Nov 03 08:06:16.470812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.468753 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.470812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.469048 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.470812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.469163 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.470812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.469214 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.470812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.469276 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftct\" (UniqueName: \"kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.470812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.469315 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.570393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.570317 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.570854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.570837 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.570983 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.570968 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftct\" (UniqueName: \"kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.571071 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.571057 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.571162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.571151 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.571370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.571357 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.571496 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.571475 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.571578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.571400 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.571724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.571675 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.573336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.573318 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.573890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.573869 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.578172 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.578148 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftct\" (UniqueName: \"kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct\") pod \"distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.683917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:16.683888 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:16.999176 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:16.999142 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52a712fa_027b_4ece_81a9_a9a7c9c8c19e.slice/crio-2f25da349b03bd7f9906cd61ae9b509c5b792006a6bf769fb0b1e6abfbcab4fc WatchSource:0}: Error finding container 2f25da349b03bd7f9906cd61ae9b509c5b792006a6bf769fb0b1e6abfbcab4fc: Status 404 returned error can't find the container with id 2f25da349b03bd7f9906cd61ae9b509c5b792006a6bf769fb0b1e6abfbcab4fc Nov 03 08:06:17.026263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.026217 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4"] Nov 03 08:06:17.049370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.049337 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" event={"ID":"52a712fa-027b-4ece-81a9-a9a7c9c8c19e","Type":"ContainerStarted","Data":"2f25da349b03bd7f9906cd61ae9b509c5b792006a6bf769fb0b1e6abfbcab4fc"} Nov 03 08:06:17.051223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.051194 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" event={"ID":"ba08e75a-59aa-49ae-97ed-ca66fe8387c6","Type":"ContainerStarted","Data":"e314caae57f9d6686808b447139ee2c6677b86bc77c2d05333a82e95389a0b9e"} Nov 03 08:06:17.051337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.051244 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" event={"ID":"ba08e75a-59aa-49ae-97ed-ca66fe8387c6","Type":"ContainerStarted","Data":"e00ffb918e3e07180aa44be55d634190fa6b169de6ced85175cf8366d416f761"} Nov 03 08:06:17.053114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.053089 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" event={"ID":"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef","Type":"ContainerStarted","Data":"f2dff48b5cf9caa812458e54c022d3272f50c200fd70a4731eb0f44311ba833a"} Nov 03 08:06:17.053214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.053136 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" event={"ID":"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef","Type":"ContainerStarted","Data":"edcfb11468cbba84e92f63ce0292b54a689c9d22b6c916da757c8a446d691ea1"} Nov 03 08:06:17.114666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.099972 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podStartSLOduration=2.099953129 podStartE2EDuration="2.099953129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:17.071248268 +0000 UTC m=+716.452699186" watchObservedRunningTime="2026-11-03 08:06:17.099953129 +0000 UTC m=+716.481404047" Nov 03 08:06:17.115335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.115285 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podStartSLOduration=2.115269597 podStartE2EDuration="2.115269597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:17.097048231 +0000 UTC m=+716.478499162" watchObservedRunningTime="2026-11-03 08:06:17.115269597 +0000 UTC m=+716.496720516" Nov 03 08:06:17.901915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.901869 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:17.919143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.918784 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 08:06:17.919143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.918849 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 08:06:17.974755 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.974693 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:17.977201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.976963 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:17.977201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:17.977005 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:18.068603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.068565 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" event={"ID":"52a712fa-027b-4ece-81a9-a9a7c9c8c19e","Type":"ContainerStarted","Data":"4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292"} Nov 03 08:06:18.162782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.161074 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podStartSLOduration=2.161055828 podStartE2EDuration="2.161055828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:18.104879037 +0000 UTC m=+717.486329955" watchObservedRunningTime="2026-11-03 08:06:18.161055828 +0000 UTC m=+717.542506745" Nov 03 08:06:18.684496 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.684466 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:18.914674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.897979 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 08:06:18.914674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.898038 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 08:06:18.975972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.975870 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:18.975972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:18.975948 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:19.969670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:19.963364 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 08:06:19.969670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:19.963430 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 08:06:19.975624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:19.975593 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:19.975802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:19.975670 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:20.984147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:20.984106 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 08:06:20.984686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:20.984170 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 08:06:20.986706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:20.986295 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:20.986930 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:20.986905 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:21.899164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:21.899129 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 08:06:21.899487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:21.899445 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 08:06:21.957114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:21.957077 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:21.975094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:21.975058 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:21.975340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:21.975317 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:21.975523 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:21.975270 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:22.174136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:22.174053 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:22.904669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:22.898042 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 08:06:22.904669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:22.898105 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 08:06:22.975182 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:22.975146 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:22.975454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:22.975414 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:23.975957 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:23.975919 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 08:06:23.976525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:23.976491 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 08:06:24.148473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:24.148446 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:24.150523 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:24.149828 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:24.151707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:24.151678 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:06:24.980794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:24.980766 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:24.981282 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:24.981051 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:24.981986 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:24.981964 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:06:26.690232 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:26.690196 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:27.451362 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.451318 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.543926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.543881 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.593790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.593738 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.640802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.640751 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.658826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.658766 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:06:27.671808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.670728 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.704725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.702744 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.706245 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.706207 5951 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:27.990703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:27.990583 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:28.011783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.011736 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:28.142986 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.142942 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:28.550670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.549742 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:28.550670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.550551 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:06:28.611671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.595730 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:28.611671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.596513 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:06:28.688670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.663746 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:06:28.717702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.712792 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:06:28.768682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.756879 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:06:28.772671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.769000 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:28.797837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.797789 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:29.023733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.936387 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:06:29.023733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.936458 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:06:29.023733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:28.936498 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:06:29.050963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.024711 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:29.050963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.035276 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:29.202280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.201747 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:06:29.556671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.549996 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:29.600812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.600764 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:06:29.672668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.667782 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:29.804162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.804126 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:06:29.897378 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.897296 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:29.922045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.922015 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:29.930694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.924513 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:30.055874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.055827 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 03 08:06:30.056140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.970156 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:06:30.056540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.970119 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:30.082672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.057950 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:06:30.111052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:29.970177 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 08:06:30.111338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.084129 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:30.218710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.218583 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:30.255817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.255763 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:30.582671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.554287 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:30.629788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.629738 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:30.693673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:30.668713 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:31.064752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.062045 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:31.064752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.063302 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:31.112667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.068320 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:31.181206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.181165 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:06:31.187854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.187814 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 08:06:31.217604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.214874 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:06:31.347752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.347626 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:06:31.525296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.525259 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:31.541688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.538300 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:06:31.552846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.552813 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:06:31.607138 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.602892 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp"] Nov 03 08:06:31.629078 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.626169 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp"] Nov 03 08:06:31.629078 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.626272 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:06:31.629078 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.627511 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.640807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.631862 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:06:31.640807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.634422 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:31.687579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.676156 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:06:31.691936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.691008 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:06:31.693929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.693906 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:06:31.695147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.695125 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:31.722733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.722697 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.722904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.722749 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkdx\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.722904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.722844 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.723002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.722900 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.723048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.722998 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.723100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.723083 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.723141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.723110 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.723192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.723164 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.723249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.723230 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.828457 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.828421 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.828738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.828708 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkdx\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.828869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.828800 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.828869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.828846 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.828993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.828908 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.829054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.829016 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.829110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.829049 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.829168 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.829137 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.829257 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.829240 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.830107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.830077 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.830353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.830329 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.830784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.830762 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.831081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.831043 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.832416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.831841 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.850929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.850893 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.853102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.853073 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.863100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.863017 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.863324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.863287 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkdx\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:31.972162 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:06:31.972123 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:06:31.974847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:31.974827 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:32.279601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.279574 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp"] Nov 03 08:06:32.280334 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:32.280305 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a52f05e_2638_4c97_ae7e_438d5a93382d.slice/crio-29d602145b5ae612f8eed9cd6fee813064e229825f4d014eb3dec7d10a236ec3 WatchSource:0}: Error finding container 29d602145b5ae612f8eed9cd6fee813064e229825f4d014eb3dec7d10a236ec3: Status 404 returned error can't find the container with id 29d602145b5ae612f8eed9cd6fee813064e229825f4d014eb3dec7d10a236ec3 Nov 03 08:06:32.283125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.283093 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:32.283212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.283160 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:32.283212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.283198 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:32.846622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.846585 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" event={"ID":"2a52f05e-2638-4c97-ae7e-438d5a93382d","Type":"ContainerStarted","Data":"898a0b2640550e79c5f947d656a900bb8259c21c21d33389a2f258c6864390d4"} Nov 03 08:06:32.846832 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.846698 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" event={"ID":"2a52f05e-2638-4c97-ae7e-438d5a93382d","Type":"ContainerStarted","Data":"29d602145b5ae612f8eed9cd6fee813064e229825f4d014eb3dec7d10a236ec3"} Nov 03 08:06:32.865100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.864741 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podStartSLOduration=2.864721166 podStartE2EDuration="2.864721166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:32.863405537 +0000 UTC m=+732.244856478" watchObservedRunningTime="2026-11-03 08:06:32.864721166 +0000 UTC m=+732.246172083" Nov 03 08:06:32.975711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:32.975678 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:33.978858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:33.977250 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 08:06:33.978858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:33.977321 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 08:06:34.976101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:34.976055 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 08:06:34.976276 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:34.976122 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 08:06:35.975932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:35.975895 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 08:06:35.976852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:35.975954 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 08:06:36.711022 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:36.710991 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:36.744066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:36.744030 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:36.976468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:36.976382 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 08:06:36.976468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:36.976447 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 08:06:37.975973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:37.975933 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 08:06:37.976161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:37.976008 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 08:06:38.976356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:38.976320 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 08:06:38.976788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:38.976385 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 08:06:40.002485 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:40.002449 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:40.004530 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:40.003907 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:40.004883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:40.004862 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:06:45.790843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.790809 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5"] Nov 03 08:06:45.809843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.809809 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.824194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.824149 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5"] Nov 03 08:06:45.830145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.830091 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz"] Nov 03 08:06:45.850889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.850858 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz"] Nov 03 08:06:45.851087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.850957 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:45.915547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.915511 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.916880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.916854 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.916886 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.916964 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnw9s\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.917014 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.917049 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.917100 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.917259 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.917356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.917277 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:45.937434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.937397 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj"] Nov 03 08:06:45.942194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.942170 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:45.971140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.971093 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj"] Nov 03 08:06:45.978619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.978586 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5"] Nov 03 08:06:45.984361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:45.984337 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.021045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.021006 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.021224 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.021160 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.021330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.021313 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.021813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.021787 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.021898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.021849 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.022013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.021997 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.022062 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.022007 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5"] Nov 03 08:06:46.024669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.023002 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.025723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.025692 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.025845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.025776 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.025907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.025847 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.025907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.025888 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.026020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.025948 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.026020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.025974 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.026020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026004 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.026463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026337 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnw9s\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.026463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026389 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.026463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026429 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.026684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026475 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fmg\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.026684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026479 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.027732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.026853 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.027732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.027075 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.027732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.027139 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.028441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.028394 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.035492 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.033354 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.036933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.036906 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.064197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.064106 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.068854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.066795 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnw9s\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.128813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.128772 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.128992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.128828 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.128992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.128895 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.128992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.128920 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.128992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.128947 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.128992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.128981 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.129272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129112 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.129272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129145 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129175 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.129435 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129315 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129435 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129316 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129435 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129389 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.129435 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129428 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129480 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.129678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129514 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129562 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.129678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129595 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129608 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129801 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.129966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129932 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsvk\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.130102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.129994 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbgtr\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.130102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130033 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.130102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130058 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.130102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130061 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.130102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130078 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.130344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130179 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.130344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130222 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.130344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130331 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fmg\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.130498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130352 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.130498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130391 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.130498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130419 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.130498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.130436 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.132090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.132067 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.132782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.132763 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.136969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.136941 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.137996 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.137974 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fmg\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.150407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.150381 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:46.169837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.169810 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.238934 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239002 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239032 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239082 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239148 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239173 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239222 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239270 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239306 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239388 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239452 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239508 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239516 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239596 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsvk\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239636 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbgtr\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239728 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239756 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239796 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.240667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.239836 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.249394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.242149 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.249394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.243282 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.249394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.247395 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.249394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.248844 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.257588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.254590 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.257588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.256970 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.257588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.257240 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.258486 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.258463 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.262481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.261927 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.262481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.262438 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.262726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.262557 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.270836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.265146 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.271994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.271967 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbgtr\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.272274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.272248 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.277238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.277171 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.277623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.277599 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsvk\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.288025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.287986 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.305852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.305557 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:46.489875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.488584 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz"] Nov 03 08:06:46.500800 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:46.494103 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88272ac9_229c_4604_96bc_e02d42140726.slice/crio-bf63cd46285e47dd29a14e8a326ee7898a4673d47f35cf448ef9c7d4aed41f53 WatchSource:0}: Error finding container bf63cd46285e47dd29a14e8a326ee7898a4673d47f35cf448ef9c7d4aed41f53: Status 404 returned error can't find the container with id bf63cd46285e47dd29a14e8a326ee7898a4673d47f35cf448ef9c7d4aed41f53 Nov 03 08:06:46.500800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.497368 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5"] Nov 03 08:06:46.500800 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:46.500470 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac2b7e5_0558_40a5_953c_1b3f57ce8918.slice/crio-d9fc53ccd5abe24b8ca1ffe5410ed3e51151bb3abb10bf4c2e1698a45ce41f0b WatchSource:0}: Error finding container d9fc53ccd5abe24b8ca1ffe5410ed3e51151bb3abb10bf4c2e1698a45ce41f0b: Status 404 returned error can't find the container with id d9fc53ccd5abe24b8ca1ffe5410ed3e51151bb3abb10bf4c2e1698a45ce41f0b Nov 03 08:06:46.505013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.504974 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.505133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.505050 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.505133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.505062 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.505133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.505083 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.505294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.505135 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.505294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.505180 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.563941 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.563908 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:46.668538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.668492 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5"] Nov 03 08:06:46.678771 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:46.678722 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c490ea_fbef_4135_a1ad_5e67d92bd9bc.slice/crio-6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704 WatchSource:0}: Error finding container 6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704: Status 404 returned error can't find the container with id 6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704 Nov 03 08:06:46.682252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.682215 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.682369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.682307 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.682369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.682341 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.931360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.931314 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj"] Nov 03 08:06:46.937625 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:06:46.936429 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60e3d10_bc4e_42c9_93af_51c843e47fd9.slice/crio-943cd52e648031d1a390fb8a4d5b696e1a10462542fa883174d6391377dc658e WatchSource:0}: Error finding container 943cd52e648031d1a390fb8a4d5b696e1a10462542fa883174d6391377dc658e: Status 404 returned error can't find the container with id 943cd52e648031d1a390fb8a4d5b696e1a10462542fa883174d6391377dc658e Nov 03 08:06:46.940498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.940465 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.940721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.940698 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:46.940878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:46.940857 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:06:47.051433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.051346 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" event={"ID":"d60e3d10-bc4e-42c9-93af-51c843e47fd9","Type":"ContainerStarted","Data":"943cd52e648031d1a390fb8a4d5b696e1a10462542fa883174d6391377dc658e"} Nov 03 08:06:47.055717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.055360 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" event={"ID":"5ac2b7e5-0558-40a5-953c-1b3f57ce8918","Type":"ContainerStarted","Data":"f7806c0114ac1f2913816750d0c8d35e7ae156af88e19b5dfa67ad48e89d0c22"} Nov 03 08:06:47.055717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.055416 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" event={"ID":"5ac2b7e5-0558-40a5-953c-1b3f57ce8918","Type":"ContainerStarted","Data":"d9fc53ccd5abe24b8ca1ffe5410ed3e51151bb3abb10bf4c2e1698a45ce41f0b"} Nov 03 08:06:47.076103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.076070 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" event={"ID":"88272ac9-229c-4604-96bc-e02d42140726","Type":"ContainerStarted","Data":"419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641"} Nov 03 08:06:47.076335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.076318 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" event={"ID":"88272ac9-229c-4604-96bc-e02d42140726","Type":"ContainerStarted","Data":"bf63cd46285e47dd29a14e8a326ee7898a4673d47f35cf448ef9c7d4aed41f53"} Nov 03 08:06:47.119664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.114656 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podStartSLOduration=2.114618076 podStartE2EDuration="2.114618076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.113929475 +0000 UTC m=+746.495380413" watchObservedRunningTime="2026-11-03 08:06:47.114618076 +0000 UTC m=+746.496068994" Nov 03 08:06:47.126421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.123498 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" event={"ID":"36c490ea-fbef-4135-a1ad-5e67d92bd9bc","Type":"ContainerStarted","Data":"355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6"} Nov 03 08:06:47.126421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.123565 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" event={"ID":"36c490ea-fbef-4135-a1ad-5e67d92bd9bc","Type":"ContainerStarted","Data":"6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704"} Nov 03 08:06:47.138692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.136194 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podStartSLOduration=2.136176278 podStartE2EDuration="2.136176278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.13311666 +0000 UTC m=+746.514567603" watchObservedRunningTime="2026-11-03 08:06:47.136176278 +0000 UTC m=+746.517627194" Nov 03 08:06:47.173667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.167321 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:47.173667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.171448 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:47.173667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.171554 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:47.173667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.171593 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:47.173667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.171922 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:47.173667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.171958 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:47.184668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.180318 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podStartSLOduration=2.180300481 podStartE2EDuration="2.180300481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.167143032 +0000 UTC m=+746.548594030" watchObservedRunningTime="2026-11-03 08:06:47.180300481 +0000 UTC m=+746.561751399" Nov 03 08:06:47.314509 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.314476 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:47.314865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.314841 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:47.315031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:47.315007 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:48.180076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.179502 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:48.180076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.179564 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:48.180076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.179989 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:48.180076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.180024 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:48.222182 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.220207 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" event={"ID":"d60e3d10-bc4e-42c9-93af-51c843e47fd9","Type":"ContainerStarted","Data":"5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba"} Nov 03 08:06:48.248299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.246358 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podStartSLOduration=3.246336702 podStartE2EDuration="3.246336702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:48.243444325 +0000 UTC m=+747.624895243" watchObservedRunningTime="2026-11-03 08:06:48.246336702 +0000 UTC m=+747.627787625" Nov 03 08:06:48.330151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.330114 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:48.330393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.330366 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:48.607671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.586817 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:48.611688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.608800 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:48.611688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:48.608865 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:49.060880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.060815 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:49.188746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.188709 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:49.189357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.189331 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:49.189862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.188732 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:49.190277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.188808 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:49.190417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.190396 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:49.281764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.280710 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:06:49.322807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.322692 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:49.323556 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.323051 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:49.496683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.489758 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:49.541103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.539973 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:49.566252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.566210 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:49.566554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.566523 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:49.611927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.611821 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:06:49.644802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.644756 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:49.734002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.733942 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:49.747813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:49.747769 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:50.059673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.029616 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:50.142668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.133287 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:50.142668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.134226 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:06:50.177671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.152282 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:50.339753 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.238173 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.040990 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.547130 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.580963 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.065911 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.383160 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.097229 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.713484 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.097320 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:50.976851 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.124671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.097373 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:51.236211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.219958 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:51.236211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.221173 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:06:51.236211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.222891 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:06:51.288843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.263086 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.288843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.263171 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:51.310672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.302896 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" podUID="9d85e397-5d01-4200-84d6-ccd89ffdb2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 08:06:51.310672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.303835 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.310672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.303881 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:51.390724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.379693 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:06:51.459916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.459861 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.460252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.460221 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:51.460856 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.460826 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:51.461034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.461012 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:51.549885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.512775 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:06:51.956799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:51.577776 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:06:52.182202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.007801 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:06:52.214937 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.115284 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:52.243921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.115327 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:52.244392 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.244199 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:06:52.245829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.149860 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:52.276804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.276767 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:52.277087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.252179 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:06:52.314269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.218707 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:52.314269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.283279 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:52.314269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.299770 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:52.314269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.313385 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 08:06:52.343686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.342732 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:52.343686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.342802 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:52.367689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.367527 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:52.367689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.367583 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:52.388590 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.388542 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:06:52.513868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.513755 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:06:52.652946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.641113 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:52.653260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.653224 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:52.872845 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:06:52.872673 5951 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.806s" Nov 03 08:06:52.872845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.872790 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:06:52.872845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.872856 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:06:52.873173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.872903 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:52.979670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:52.963872 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.025788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.025744 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.063185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.063139 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.171728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.152024 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:53.172020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.171978 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:53.172167 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.167743 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.176183 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.176149 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:53.176959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.176923 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:53.188684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.188608 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:06:53.227898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.227851 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:06:53.306286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.244222 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.306286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.293874 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.329241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.323292 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:06:53.360671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.332275 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:53.566166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.337877 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:53.582527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.403797 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.583063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.480792 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:53.583318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.583300 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:06:53.583471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.583460 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:06:53.636021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.594555 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:53.636021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.594605 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:06:53.636021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.594622 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:06:53.636021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.594668 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:06:53.636021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.538756 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:53.637549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.637158 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:53.822279 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.798989 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:53.822279 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:53.799051 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:54.217760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.210810 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:06:54.217760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.211620 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:06:54.242734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.219993 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:54.242734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.220056 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:54.242734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.237850 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:54.291697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.245878 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:54.320694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.295636 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:54.348705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.341144 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:54.348705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.341207 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:54.348705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.341363 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:06:54.387775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.379495 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:54.387775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.379571 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:54.549334 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.549278 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:54.597676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.583747 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:54.597676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.584137 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:54.623673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.608711 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:54.695848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.695811 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:54.696088 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:54.696062 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:55.117275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.117215 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.295786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.221363 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:55.296442 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.296405 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.249324 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.322491 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.222833 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.319094 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.230468 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.323796 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.320603 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.323862 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.342189 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.355532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.342971 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.484837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.275622 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:06:55.616992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.616954 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:55.617630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.617597 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:55.619256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.619211 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.670023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.669975 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:55.671682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.671603 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:06:55.673067 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.673012 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:06:55.692911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.692870 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:06:55.693100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:55.692963 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:06:56.326680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.324050 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:56.348864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.327011 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:06:56.348864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.338850 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 08:06:56.348864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.338910 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 08:06:56.348864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.347060 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:56.348864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.347127 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:56.391840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.391803 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 08:06:56.430811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.419822 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 08:06:56.473805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.473762 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:56.507254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.505378 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:06:56.527865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.526336 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:06:56.590769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.561358 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:06:56.591699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.591620 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:06:56.654052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.654012 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:56.669254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.669210 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:56.683690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.681977 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:56.683690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.682055 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:56.718249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.718206 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:06:56.992932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.992848 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:06:56.993434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.993412 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:06:56.993609 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.993595 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:06:56.993808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.993792 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:06:56.993949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.993936 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:06:56.994079 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:56.994066 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:06:57.310367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.310331 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 08:06:57.310683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.310628 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 08:06:57.568270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.567735 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 08:06:57.568270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.567801 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 08:06:57.782670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.772979 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:57.782670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.774561 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:57.786668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:57.784240 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:06:58.032003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.031969 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:58.036090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.036064 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:58.041862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.041834 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:06:58.570194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.570161 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:58.570588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.570566 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:58.571416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.571399 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:06:58.637064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.637027 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:58.639580 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.639556 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:06:58.643014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:06:58.642994 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:07:01.746679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:01.731415 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:01.978475 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:01.978376 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.057812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.042273 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.190669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.189804 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.220795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.220747 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:02.285668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.281057 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:02.285668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.281780 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:02.285668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.282964 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.317678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.288022 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.318251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.303995 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.506665 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:07:02.506224 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:07:02.612439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.612333 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:02.781085 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.763771 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.984953 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.984826 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:02.986036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:02.985990 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Nov 03 08:07:03.045671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.043366 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.045671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.044882 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.061671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.051340 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:07:03.061671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.051483 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:07:03.061671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.051523 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:07:03.223035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.222998 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.569787 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.245483 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.388559 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.580382 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.580449 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.389610 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.388520 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.610907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.389583 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.680069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.633813 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:07:03.680069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.633927 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:07:03.680069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.633965 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:07:03.856822 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.780813 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.856822 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.781214 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:07:03.856822 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.840914 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:03.998800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:03.998693 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:04.045037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.043746 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:04.568066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.568010 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:04.570995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.570953 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:04.579752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.579710 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:04.617355 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.617296 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:07:04.620103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.620064 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:04.649787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.649724 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:04.787818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:04.787761 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:05.020077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.019959 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:05.054898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.035790 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:05.110675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.056173 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Nov 03 08:07:05.279631 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.056191 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:07:05.280174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.097583 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:05.594733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.568668 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:05.596120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.582872 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:05.597176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.584359 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:05.642808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.632389 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:07:05.642808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.633154 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:05.683764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.672856 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:05.866164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:05.866022 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:06.100737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.100684 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:06.101828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.101793 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:06.286174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.284156 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:07:06.318441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.318407 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:07:06.326278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.326206 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:07:06.366248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.366201 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:07:06.367327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.367300 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:07:06.386056 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.386025 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:07:06.425789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.425759 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:07:06.468423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.468238 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:07:06.557672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.556126 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:07:06.560792 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:06.560768 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:07:27.479669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.473760 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:27.574828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.574769 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:27.614780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.614728 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:27.615400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.615038 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:27.633811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.633735 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:27.662672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.661734 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:27.735885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.735770 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:27.793804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.793756 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:27.850849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:27.850798 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:28.048670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.042197 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:28.079671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.078760 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:28.534852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.534804 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:28.536104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.536077 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:07:28.642673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.639240 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:28.651684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.639185 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:28.651684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.645488 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:07:28.651684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.646713 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:28.651684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.647940 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:07:28.651684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.647996 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:07:28.677673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.646107 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:07:28.677673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.661739 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:07:28.678798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.678738 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:28.708696 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.699574 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:07:28.934775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:28.933717 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:29.092671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.085107 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.092671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.086180 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.183673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.182023 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.269021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.268927 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:07:29.269353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.269334 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:07:29.269555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.269541 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:07:29.269726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.269713 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:07:29.570145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.544631 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.646671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.645803 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.650682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.646996 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:29.650682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.647636 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.650682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.648210 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:29.650682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.648245 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.669674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.664390 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.701165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.700751 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.796671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.793243 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.853863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.853771 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:29.854151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.854122 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:29.941788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:29.941735 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:30.208725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.126340 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:30.208725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.126936 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:07:30.208725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.169847 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:30.208725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.184992 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:30.595670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.595383 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:07:30.658980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.653790 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:30.663494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.660181 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:30.663494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.661723 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:30.670802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.670736 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:30.714673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.703258 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:30.990357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:30.969687 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 08:07:31.322708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.322625 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:31.323902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.323869 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:31.356709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.354536 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:31.356709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.356495 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:07:31.723915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.723791 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:31.751138 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.751090 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:31.784861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.784811 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:31.906320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.906261 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:31.999811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.913354 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:07:31.999811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:31.954913 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:32.410691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.409056 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:32.410691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.410486 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:32.436672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.411619 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:32.436672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.427734 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:32.456294 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:07:32.456235 5951 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.709s" Nov 03 08:07:32.564210 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.564163 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8"] Nov 03 08:07:32.760733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.733733 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:32.760733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.754074 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:32.771599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.771563 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8"] Nov 03 08:07:32.820767 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.778692 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:32.994724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:32.994668 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz"] Nov 03 08:07:33.028877 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.028836 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz"] Nov 03 08:07:33.029280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.029257 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037582 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037674 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037722 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037782 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037838 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037868 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037914 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zst52\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.037977 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038044 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038123 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvk6\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038170 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038205 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038246 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038271 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038331 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038363 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038396 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.054705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.038423 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.162453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.162411 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.231105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.231071 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.231404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.231373 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.231677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.231564 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zst52\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.231978 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.231956 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.232325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.232152 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.232697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.163589 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.232963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.232806 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvk6\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.237804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.237769 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.246870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.246805 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.247218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.247193 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.247385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.247366 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.254754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.250805 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.258755 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.258695 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.263788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.263740 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.264134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.264099 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.278860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.276930 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.290943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.277501 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.294830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.235286 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.295092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.277844 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.295240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.258974 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.413620 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:07:33.180778 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:07:33.413901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.235906 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.414038 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.278524 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.414158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.279177 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.414327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.414307 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.415037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.415011 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.415338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.290789 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.415464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.356252 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.415574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.410885 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.431704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.290722 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.431704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.412457 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.431704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.421511 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.431704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.410928 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.431704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.283955 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.434873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.432880 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.434873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.433710 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.458832 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.458315 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.500772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.500723 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.512879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.512719 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zst52\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:33.561703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.561525 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 03 08:07:33.649307 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.639263 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:33.649626 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.647827 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.683783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.674618 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:33.683783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.676046 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 08:07:33.741186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.739347 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.762072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.760796 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:33.861104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:33.857471 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvk6\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:34.073545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.073459 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:34.130064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.113127 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:34.268203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.267881 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:34.416121 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.416038 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:34.444037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.435160 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:34.451342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.435195 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:34.451549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.437046 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:07:34.452075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.437077 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:34.452075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.437096 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:34.523816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.521768 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:34.682636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.677073 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:34.682636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.678559 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:34.739857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.739815 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:34.773667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:34.761353 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:35.029123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.008768 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:35.286790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.283480 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:07:35.337896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.332421 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:07:35.337896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.332508 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:07:35.337896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.332559 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:07:35.337896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.333260 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:07:35.337896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.336011 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:07:35.344638 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.344605 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:07:35.361832 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.349389 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:07:35.361832 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.354297 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:07:35.388754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.388726 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:07:35.391920 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.391891 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8"] Nov 03 08:07:35.392394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.392359 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:35.392579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.392557 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:35.392773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.392754 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:35.766974 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.766937 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz"] Nov 03 08:07:35.771871 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.771826 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:35.771979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.771959 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:35.772026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:35.772007 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:36.386290 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.386229 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" event={"ID":"e27fc31e-f736-47f6-b5c2-f1e3e948f44e","Type":"ContainerStarted","Data":"696fcc7549ed0b41be506258c68b79e1001ccf699ce10d80e347f2a538bd3c30"} Nov 03 08:07:36.386791 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.386342 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" event={"ID":"e27fc31e-f736-47f6-b5c2-f1e3e948f44e","Type":"ContainerStarted","Data":"491d815322de5b90164763ec78ee34a7c944289bcfb3ae26333c1bb3a4375dd0"} Nov 03 08:07:36.389212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.389174 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" event={"ID":"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3","Type":"ContainerStarted","Data":"9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8"} Nov 03 08:07:36.389326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.389233 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" event={"ID":"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3","Type":"ContainerStarted","Data":"2aeb12c0735bd0b4f870a331c6b62775033b6e4f4591e698332ff082f7f6801c"} Nov 03 08:07:36.407067 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.406938 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podStartSLOduration=6.406919192 podStartE2EDuration="6.406919192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:36.404427347 +0000 UTC m=+795.785878265" watchObservedRunningTime="2026-11-03 08:07:36.406919192 +0000 UTC m=+795.788370110" Nov 03 08:07:36.427473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.427406 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podStartSLOduration=6.4273883099999995 podStartE2EDuration="6.42738831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:36.424158424 +0000 UTC m=+795.805609342" watchObservedRunningTime="2026-11-03 08:07:36.42738831 +0000 UTC m=+795.808839230" Nov 03 08:07:36.440693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.440614 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:36.440943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.440918 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:36.441290 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:36.441271 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:37.279738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:37.278380 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:37.279738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:37.278555 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:37.279738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:37.278603 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:37.438673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:37.438327 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:37.438673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:37.438387 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:38.275273 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:38.275234 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:38.275499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:38.275303 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:38.436324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:38.436277 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:38.436525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:38.436345 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:39.275712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:39.275681 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:39.276180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:39.276154 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:39.448718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:39.444336 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:39.448718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:39.444416 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:40.276789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:40.276746 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:40.277310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:40.276811 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:40.437670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:40.436856 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:40.437670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:40.436936 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:41.275027 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:41.274993 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:41.275227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:41.275083 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:41.436136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:41.436096 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:41.438000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:41.437963 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:42.279629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:42.279588 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:42.279968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:42.279937 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:42.443736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:42.435522 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:42.443736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:42.435580 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:43.275435 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:43.275391 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 08:07:43.275697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:43.275467 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 08:07:43.435787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:43.435744 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 08:07:43.435996 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:43.435812 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 08:07:44.280240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:44.280209 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:44.280773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:44.280614 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:44.281479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:44.281458 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:07:44.441388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:44.441354 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:44.441923 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:44.441898 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:44.443280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:44.443260 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:07:47.424810 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.424742 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:07:47.449688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.439835 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.449688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.440232 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:47.464169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.463206 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.690475 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.690370 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.712787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.712743 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.730672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.713379 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.756117 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.756073 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.791048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.784628 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.947857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.947752 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:47.948452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:47.948419 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:48.206222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.206114 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.430657 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.430573 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.431782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.431760 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:07:48.456216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.456162 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.458015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.456570 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:07:48.464100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.464050 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.517592 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.517474 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:07:48.695467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.693778 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:48.704735 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.695822 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:07:48.773234 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.773133 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:07:48.791401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.781776 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.791401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.788102 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.967216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.964202 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:48.967216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:48.964552 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:07:49.182769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.040122 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:07:49.182769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.115316 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:07:49.182769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.115395 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:07:49.182769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.115455 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:07:49.182769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.115510 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:07:49.214252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.214193 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:49.217178 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.217151 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:07:49.435868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.435277 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:49.549672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.519394 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:49.715764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.542844 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:49.738675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.736270 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:49.958672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.916852 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.058922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.918839 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.094924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.918881 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.146701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.918905 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:50.286671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.968590 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:50.286671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:49.968658 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:07:50.286671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.038234 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.286671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.054741 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.286671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.285198 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:50.365736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.363758 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.485671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.460106 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:07:50.635771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.629995 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.650680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.650225 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:50.759670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:50.759345 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.037275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.025328 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 08:07:51.061813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.061752 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.290737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.274726 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.290737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.278177 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.303093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.303047 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.304710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.304668 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.345476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.345428 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.467082 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.467035 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.718728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.653701 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.718728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.654799 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:51.834676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:51.784282 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.097074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.097026 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.282603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.282560 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.283577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.283529 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:07:52.322891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.322841 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.323367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.323333 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.323627 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.323592 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:52.350307 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.350196 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.491811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.491755 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.660081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.659961 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.662918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.660334 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.786514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.786467 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:52.967741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.938790 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9"] Nov 03 08:07:52.970814 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.970769 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:52.990681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.978984 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\"" Nov 03 08:07:52.992689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.990987 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Nov 03 08:07:53.000011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:52.999973 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 08:07:53.014805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.006938 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.023896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.015098 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.023896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.015147 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nht\" (UniqueName: \"kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.050270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.039083 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:53.050270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.039147 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.087817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.059692 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9"] Nov 03 08:07:53.116669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.116496 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.116669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.116595 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.141668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.122788 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.141668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.122851 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nht\" (UniqueName: \"kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.141668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.123235 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.141668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.135212 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.255730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.254886 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:53.255730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.254967 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.320677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.320287 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.346979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.327822 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:53.346979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.329925 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:53.346979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.339992 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.396386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.364815 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.495805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.495748 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.674670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.663065 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.704118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.675069 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:53.762672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.747083 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nht\" (UniqueName: \"kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:53.787955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:53.787912 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:54.148764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.090188 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89f6363-xnqb9\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:54.148764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.103634 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:07:54.202807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.202757 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:54.361678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.354547 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:54.361678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.355753 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:07:54.381308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.374053 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:54.381308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.376741 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:54.381308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.377590 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:54.594676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.586419 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:54.778730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.718838 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:07:54.864236 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.862683 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:54.913999 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:54.870794 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:55.009754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.008944 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Nov 03 08:07:55.260012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.234032 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.260012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.250939 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:55.260012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.250998 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.281721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.272358 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:55.281721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.272421 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.314393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.293754 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:55.314393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.293824 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.360675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.336014 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:55.360675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.336101 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.504188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.469734 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:07:55.504188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.471167 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.504188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.471439 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.541926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.532823 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:55.592117 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.554730 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:07:55.628139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.617274 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.739706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.729342 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.739706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.731284 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:55.739706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.731342 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.843231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.842699 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:55.843231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.842768 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:55.908740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.908692 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:07:55.919754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:55.918926 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:07:56.068917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.068873 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:07:56.069142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.068971 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.344669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.294232 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:07:56.491708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.472085 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.491708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.472510 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.492336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.492304 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.619762 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.618376 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.668663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.667519 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.668998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.668962 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:07:56.697805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.692973 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:07:56.704335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.704073 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:07:56.710904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.710880 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:07:56.739109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.738870 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:07:56.806534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.805893 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:07:56.806534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.805955 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:07:56.806534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.805988 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:07:56.806534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.806019 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:07:56.811451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.811415 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:07:56.834329 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.834302 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:07:56.843341 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.843309 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:07:56.954627 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:07:56.954581 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aae1003_32cf_4467_8151_4fa91cc611af.slice/crio-a2156b95508c442894a933c5d1d3e0bb82c9e3ca4ba7087f4eb96789fd3427ed WatchSource:0}: Error finding container a2156b95508c442894a933c5d1d3e0bb82c9e3ca4ba7087f4eb96789fd3427ed: Status 404 returned error can't find the container with id a2156b95508c442894a933c5d1d3e0bb82c9e3ca4ba7087f4eb96789fd3427ed Nov 03 08:07:56.961891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:56.960064 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9"] Nov 03 08:07:57.709014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:57.708913 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" event={"ID":"7aae1003-32cf-4467-8151-4fa91cc611af","Type":"ContainerStarted","Data":"780119f888af1d1548151df330b94ee837971737b3587552f4c43f0fc067af72"} Nov 03 08:07:57.709014 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:57.708972 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" event={"ID":"7aae1003-32cf-4467-8151-4fa91cc611af","Type":"ContainerStarted","Data":"a2156b95508c442894a933c5d1d3e0bb82c9e3ca4ba7087f4eb96789fd3427ed"} Nov 03 08:07:57.725281 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:57.725227 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" podStartSLOduration=5.725207295 podStartE2EDuration="5.725207295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:57.722030048 +0000 UTC m=+817.103480965" watchObservedRunningTime="2026-11-03 08:07:57.725207295 +0000 UTC m=+817.106658213" Nov 03 08:07:58.719525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:58.719491 5951 generic.go:356] "Generic (PLEG): container finished" podID="7aae1003-32cf-4467-8151-4fa91cc611af" containerID="780119f888af1d1548151df330b94ee837971737b3587552f4c43f0fc067af72" exitCode=0 Nov 03 08:07:58.720095 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:58.719580 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" event={"ID":"7aae1003-32cf-4467-8151-4fa91cc611af","Type":"ContainerDied","Data":"780119f888af1d1548151df330b94ee837971737b3587552f4c43f0fc067af72"} Nov 03 08:07:59.993886 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:07:59.993805 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:08:00.019271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.019228 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca\") pod \"7aae1003-32cf-4467-8151-4fa91cc611af\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " Nov 03 08:08:00.019591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.019570 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht\" (UniqueName: \"kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht\") pod \"7aae1003-32cf-4467-8151-4fa91cc611af\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " Nov 03 08:08:00.020044 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.020017 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca\") pod \"7aae1003-32cf-4467-8151-4fa91cc611af\" (UID: \"7aae1003-32cf-4467-8151-4fa91cc611af\") " Nov 03 08:08:00.025737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.021005 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca" pod "7aae1003-32cf-4467-8151-4fa91cc611af" (UID: "7aae1003-32cf-4467-8151-4fa91cc611af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:08:00.026698 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.026663 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca" pod "7aae1003-32cf-4467-8151-4fa91cc611af" (UID: "7aae1003-32cf-4467-8151-4fa91cc611af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:08:00.031192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.026909 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-openshift-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:08:00.033195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.033082 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht" pod "7aae1003-32cf-4467-8151-4fa91cc611af" (UID: "7aae1003-32cf-4467-8151-4fa91cc611af"). InnerVolumeSpecName "kube-api-access-q6nht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:08:00.133147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.133104 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aae1003-32cf-4467-8151-4fa91cc611af-maas-api-xks-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:08:00.133147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.133146 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6nht\" (UniqueName: \"kubernetes.io/projected/7aae1003-32cf-4467-8151-4fa91cc611af-kube-api-access-q6nht\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:08:00.747353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.747257 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" event={"ID":"7aae1003-32cf-4467-8151-4fa91cc611af","Type":"ContainerDied","Data":"a2156b95508c442894a933c5d1d3e0bb82c9e3ca4ba7087f4eb96789fd3427ed"} Nov 03 08:08:00.747353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.747302 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9" Nov 03 08:08:00.747353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:00.747315 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2156b95508c442894a933c5d1d3e0bb82c9e3ca4ba7087f4eb96789fd3427ed" Nov 03 08:08:01.191529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.191494 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6"] Nov 03 08:08:01.192238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.192223 5951 cpu_manager.go:414] "containerMap: removing container" podUID="7aae1003-32cf-4467-8151-4fa91cc611af" containerName="revoke-keys" Nov 03 08:08:01.192288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.192239 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aae1003-32cf-4467-8151-4fa91cc611af" containerName="revoke-keys" Nov 03 08:08:01.192380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.192371 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aae1003-32cf-4467-8151-4fa91cc611af" containerName="revoke-keys" Nov 03 08:08:01.201236 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.201204 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.213448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.212602 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6"] Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.254955 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255187 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k24n\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255239 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255324 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255359 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255457 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255494 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255529 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.256734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.255610 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.357667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.357614 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.357844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.357700 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.357888 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.357872 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k24n\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.357936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.357917 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.357996 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358039 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358126 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358166 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358261 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358200 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358363 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358340 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358375 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358492 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358463 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358533 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.358892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.358868 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.360699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.360609 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.360819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.360719 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.365059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.365024 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.365191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.365115 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k24n\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.553029 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.552993 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:01.859338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.859303 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6"] Nov 03 08:08:01.860806 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:08:01.860776 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043d6408_4f88_456b_a6e8_524a42188208.slice/crio-cd186d2f1a3044a744031ee4e32598862e2f67979804689d3ceb0f360c503117 WatchSource:0}: Error finding container cd186d2f1a3044a744031ee4e32598862e2f67979804689d3ceb0f360c503117: Status 404 returned error can't find the container with id cd186d2f1a3044a744031ee4e32598862e2f67979804689d3ceb0f360c503117 Nov 03 08:08:01.863844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.863762 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:01.863952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.863889 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:01.863952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:01.863923 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:01.979849 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:08:01.979812 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:08:02.796591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:02.796543 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" event={"ID":"043d6408-4f88-456b-a6e8-524a42188208","Type":"ContainerStarted","Data":"26103917c5b748bd59c3b1ce3721d1dce12633e807a0ee46b2876a1531dcd2a0"} Nov 03 08:08:02.797171 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:02.796635 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" event={"ID":"043d6408-4f88-456b-a6e8-524a42188208","Type":"ContainerStarted","Data":"cd186d2f1a3044a744031ee4e32598862e2f67979804689d3ceb0f360c503117"} Nov 03 08:08:02.834666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:02.830514 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podStartSLOduration=1.830492819 podStartE2EDuration="1.830492819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:02.819173469 +0000 UTC m=+822.200624387" watchObservedRunningTime="2026-11-03 08:08:02.830492819 +0000 UTC m=+822.211943756" Nov 03 08:08:03.553685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:03.553625 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:03.554916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:03.554885 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 08:08:03.555056 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:03.554952 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 08:08:04.553509 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:04.553465 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 08:08:04.553998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:04.553534 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 08:08:05.553674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:05.553612 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 08:08:05.554485 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:05.554452 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 08:08:06.560024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.558150 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 08:08:06.560024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.558229 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 08:08:06.768635 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.768593 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm"] Nov 03 08:08:06.773405 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.773378 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.779155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.779125 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c855f06a-openshift-default-dockercfg-cjg44\"" Nov 03 08:08:06.786262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.786232 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm"] Nov 03 08:08:06.811264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811185 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811264 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811304 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811334 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5jh\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811572 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811623 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811742 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811798 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.811849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.811832 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.880712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.879236 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk"] Nov 03 08:08:06.893031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.889897 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917405 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917463 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917544 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917582 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917609 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917678 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917775 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917817 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917849 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917918 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.917995 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918027 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918076 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918106 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918145 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918177 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5jh\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918201 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918231 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jn6\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:06.926725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.918810 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.933035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.933002 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.935535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.934061 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.935806 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.934585 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.935980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.935443 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.939968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.939922 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk"] Nov 03 08:08:06.962966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.962405 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0a9dd59a-openshift-default-dockercfg-6kpkg\"" Nov 03 08:08:06.963577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.963548 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.963941 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.963841 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.966424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.966360 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:06.966725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:06.966696 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5jh\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh\") pod \"e2e-ait-c855f06a-openshift-default-978547d7c-p85lm\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:07.178726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.178003 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181421 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181538 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181588 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181656 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181685 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jn6\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181828 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181861 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181960 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.181994 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.182862 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.185426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.183204 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.227332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.202608 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.227332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.206526 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.227332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.210224 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.227332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.210515 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.227332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.210814 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.273524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.273172 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.277407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.277357 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jn6\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6\") pod \"e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.392164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.392069 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:07.569678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.568943 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 08:08:07.569678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.569003 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 08:08:07.839252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.839201 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm"] Nov 03 08:08:07.864660 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.864577 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:07.864860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.864829 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:07.872377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:07.869830 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:08.060491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.060427 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" event={"ID":"424a6ef4-dfba-484d-b43f-481d3b09901d","Type":"ContainerStarted","Data":"e36782d92dfd919c1b2809368816c26e99131c3a1c45819f3306f1d068e20556"} Nov 03 08:08:08.202754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.200313 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk"] Nov 03 08:08:08.239804 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:08:08.239764 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9d7b52_fe82_49f0_a609_1493cf8020fc.slice/crio-f52a0dff0e61549de1832715b56469c806651c6a102bdb0ed89dc81a507ef5d4 WatchSource:0}: Error finding container f52a0dff0e61549de1832715b56469c806651c6a102bdb0ed89dc81a507ef5d4: Status 404 returned error can't find the container with id f52a0dff0e61549de1832715b56469c806651c6a102bdb0ed89dc81a507ef5d4 Nov 03 08:08:08.242504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.242461 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:08.242620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.242537 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:08.242620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.242576 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:08.553892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.553796 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 08:08:08.553892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:08.553860 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 08:08:09.076982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.076936 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" event={"ID":"be9d7b52-fe82-49f0-a609-1493cf8020fc","Type":"ContainerStarted","Data":"dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79"} Nov 03 08:08:09.077476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.077012 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" event={"ID":"be9d7b52-fe82-49f0-a609-1493cf8020fc","Type":"ContainerStarted","Data":"f52a0dff0e61549de1832715b56469c806651c6a102bdb0ed89dc81a507ef5d4"} Nov 03 08:08:09.079029 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.078994 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" event={"ID":"424a6ef4-dfba-484d-b43f-481d3b09901d","Type":"ContainerStarted","Data":"bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f"} Nov 03 08:08:09.099713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.099611 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" podStartSLOduration=3.099592787 podStartE2EDuration="3.099592787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:09.094671182 +0000 UTC m=+828.476122102" watchObservedRunningTime="2026-11-03 08:08:09.099592787 +0000 UTC m=+828.481043705" Nov 03 08:08:09.124713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.123493 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podStartSLOduration=3.123471888 podStartE2EDuration="3.123471888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:09.116944646 +0000 UTC m=+828.498395564" watchObservedRunningTime="2026-11-03 08:08:09.123471888 +0000 UTC m=+828.504922825" Nov 03 08:08:09.220778 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.220746 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:09.226944 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.226900 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:09.393470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.393383 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:09.400803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:09.400776 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:10.027274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.027229 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:10.028863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.028829 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:10.029840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.029815 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:08:10.466795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.466685 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:10.467398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.467378 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:10.472386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.470793 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:08:10.472386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:10.472091 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:08:12.473669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:12.472787 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:12.673924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:12.660327 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:12.673924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:12.667796 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:12.765989 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:12.765882 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:12.977672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:12.973031 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Nov 03 08:08:13.013667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.008838 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.112893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.112798 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.140667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.140595 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.330669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.330433 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:13.330879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.330850 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:13.395721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.372801 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.395721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.382582 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.495672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.488750 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.495672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.489407 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:08:13.675142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.669808 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:13.696673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.694960 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:08:13.719956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.707985 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:13.751671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.748060 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:08:13.823619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.763103 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4"] Nov 03 08:08:13.823619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.804296 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:13.906980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.906944 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:08:13.908393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.908366 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:13.947759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.945008 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0a9dd59a\"" Nov 03 08:08:13.947759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:13.945151 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0a9dd59a-dockercfg-55d7l\"" Nov 03 08:08:14.068752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.045962 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4"] Nov 03 08:08:14.202201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.144302 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:14.304669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.299867 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvq4f\" (UniqueName: \"kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f\") pod \"payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:14.304669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.300239 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:14.317121 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.311732 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:08:14.362834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.362780 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:14.399742 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.383894 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:14.496194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.489948 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:08:14.585670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.584404 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:08:14.585670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.521757 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:14.609707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.538978 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:14.823670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.588993 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:14.823670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.813131 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:14.823670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.672686 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:14.823670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.713296 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:14.855668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.842454 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:08:14.874094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.874062 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:08:14.955670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.715946 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:15.032668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.714610 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:15.032668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.556301 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:15.032668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:14.764909 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:15.054261 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.011995 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:15.056985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.032003 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:15.374669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.373621 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:15.459668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.455493 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:15.459668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.455677 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:15.600669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.594180 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:15.732736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.712435 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvq4f\" (UniqueName: \"kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f\") pod \"payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:15.866865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.866803 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:15.867738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.867702 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:15.869433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.869408 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:08:15.891033 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.890989 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:15.891493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:15.891466 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:16.159729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.158872 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:16.159729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.159209 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:16.327672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.300657 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvq4f\" (UniqueName: \"kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f\") pod \"payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:16.553995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.553953 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:16.582535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.582493 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:16.670671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.645725 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:16.670671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.666734 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:16.696028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.672800 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:16.696028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.672885 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:16.903672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.891967 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:16.904406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.904364 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:16.909293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.909255 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:16.927972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.909530 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:17.014673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:16.990282 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv"] Nov 03 08:08:17.233819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.190973 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.233819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.191373 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:17.443671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.435531 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.477277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.477243 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv"] Nov 03 08:08:17.477669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.477577 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.533731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.514268 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq"] Nov 03 08:08:17.690671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.663852 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.690671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.664379 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.690671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.664977 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:17.690671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.669358 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.690671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.670340 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c855f06a\"" Nov 03 08:08:17.690671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.670937 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c855f06a-dockercfg-nk4k9\"" Nov 03 08:08:17.749672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.737978 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq"] Nov 03 08:08:17.825958 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.758767 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.864829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.759502 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:17.872978 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.801530 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cslwp\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.874548 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.874503 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.875209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.875093 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.875549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.875529 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.875727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.875710 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.875921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.875888 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume\") pod \"payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:17.876071 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.876054 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tfw\" (UniqueName: \"kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw\") pod \"payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:17.876259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.876219 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.876416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.876399 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.876606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.876590 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.876944 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.876924 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:17.913673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.907362 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.916773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.916727 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:17.935388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.935344 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:17.967672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.936109 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:17.999324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.999278 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.023852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:17.999727 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cslwp\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.060666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.026193 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.156702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.131116 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.157066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157038 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.157225 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157209 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.157376 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157359 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume\") pod \"payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:18.157508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157491 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tfw\" (UniqueName: \"kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw\") pod \"payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:18.157673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157634 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.157820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157806 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.188668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.157945 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.189114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.066739 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cslwp\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.210328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.210266 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:18.212764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.212715 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:18.297093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.147982 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.297093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.006505 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.297093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.265607 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.297093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.133283 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.297093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.266505 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume\") pod \"payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:18.297093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.266812 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.522169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.522136 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.561678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.552250 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:18.709373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.709269 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:18.722984 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.712801 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:18.723614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.723576 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:18.962874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:18.962753 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:19.005757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.001783 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:19.005757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.003118 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:19.053353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.053290 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:19.222735 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.206753 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tfw\" (UniqueName: \"kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw\") pod \"payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:19.296480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.296408 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:19.389388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.366999 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:19.389388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.367520 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:19.416694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.406621 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:19.561669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.561371 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:19.562364 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.561778 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:19.765684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.756800 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:08:19.765684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.757258 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:19.821209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.820911 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:19.860670 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:08:19.823096 5951 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.081s" Nov 03 08:08:19.890669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.863122 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4"] Nov 03 08:08:20.428701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.428660 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" event={"ID":"7ee80e0b-21ba-4435-8680-b9fbf3ed1261","Type":"ContainerStarted","Data":"0aec5313a51c89d2796c6eb1d97cf4ba3d07e8a37f84630e8155a3b189618e17"} Nov 03 08:08:20.428976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.978113 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:20.429387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:19.924332 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:20.460671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.013942 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:20.482718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.130852 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:20.483354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.361478 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:20.483551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.413770 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:20.484209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.413835 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:20.728823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.413858 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:20.728823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.361401 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:20.940015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:20.939969 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.127678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.101050 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 08:08:21.128118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.128065 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99" podUID="9d85e397-5d01-4200-84d6-ccd89ffdb2a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.131850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.117574 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:21.159677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.158111 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m2gsg" podUID="797507d8-79e2-49b2-a51c-e72190a2fd67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 03 08:08:21.159677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.133810 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9rmr4" podUID="6cf247f2-4a20-4d4d-b55f-5fb8cd90b354" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.262749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.117517 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:08:21.429531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.429434 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.616669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.482898 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.616669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.517318 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.661110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.660808 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:21.681694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.675113 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:21.819733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.774470 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:21.819733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.776300 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:21.819733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:21.776609 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:22.264783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.243803 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:22.458729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.457792 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:08:22.458729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.458471 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:22.563792 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:08:22.563450 5951 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.624s" Nov 03 08:08:22.624505 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.507852 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:22.692715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.622296 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 08:08:22.838670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.811176 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:22.910715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.811007 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:22.932026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.931126 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:22.968366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.643749 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:22.970180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.970139 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:22.993779 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:22.993063 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:23.055740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.055707 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:08:23.056749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.056637 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:08:23.056896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.056795 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:08:23.056896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.056828 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:08:23.057159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.057141 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:08:23.057572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.057548 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:08:23.060331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.060267 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:08:23.060331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.060324 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:08:23.072249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.072222 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:08:23.074578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.074552 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:08:23.269157 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:08:23.269114 5951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee80e0b_21ba_4435_8680_b9fbf3ed1261.slice/crio-b085533f82c0ef462232d343c3f78b8cf0f8a9917fe18d25a6fc8995e34cfb1a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 08:08:23.459523 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:08:23.459474 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc18256b_bc31_42da_a928_9e468c2ba641.slice/crio-71e29e3ba49d095b09073d93357ebaeb5b6fac78df43eb197ad688fa7801fefd WatchSource:0}: Error finding container 71e29e3ba49d095b09073d93357ebaeb5b6fac78df43eb197ad688fa7801fefd: Status 404 returned error can't find the container with id 71e29e3ba49d095b09073d93357ebaeb5b6fac78df43eb197ad688fa7801fefd Nov 03 08:08:23.460119 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.460091 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv"] Nov 03 08:08:23.462409 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.462377 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:23.462507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.462465 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:23.462507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.462497 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:23.541670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.541624 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq"] Nov 03 08:08:23.546972 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:08:23.546924 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52311042_57c3_41d3_a6bc_cb01768eb10d.slice/crio-85c6282bad0f338948b9e3b29fa432eaf7d5f3a87d7fd72c385a282fedaec349 WatchSource:0}: Error finding container 85c6282bad0f338948b9e3b29fa432eaf7d5f3a87d7fd72c385a282fedaec349: Status 404 returned error can't find the container with id 85c6282bad0f338948b9e3b29fa432eaf7d5f3a87d7fd72c385a282fedaec349 Nov 03 08:08:23.723312 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.723276 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" event={"ID":"7ee80e0b-21ba-4435-8680-b9fbf3ed1261","Type":"ContainerStarted","Data":"b085533f82c0ef462232d343c3f78b8cf0f8a9917fe18d25a6fc8995e34cfb1a"} Nov 03 08:08:23.724339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.724316 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:23.728419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.728387 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" event={"ID":"52311042-57c3-41d3-a6bc-cb01768eb10d","Type":"ContainerStarted","Data":"6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c"} Nov 03 08:08:23.728555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.728458 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:23.728555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.728483 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" event={"ID":"52311042-57c3-41d3-a6bc-cb01768eb10d","Type":"ContainerStarted","Data":"85c6282bad0f338948b9e3b29fa432eaf7d5f3a87d7fd72c385a282fedaec349"} Nov 03 08:08:23.731206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.731179 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" event={"ID":"bc18256b-bc31-42da-a928-9e468c2ba641","Type":"ContainerStarted","Data":"0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4"} Nov 03 08:08:23.731330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.731228 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" event={"ID":"bc18256b-bc31-42da-a928-9e468c2ba641","Type":"ContainerStarted","Data":"71e29e3ba49d095b09073d93357ebaeb5b6fac78df43eb197ad688fa7801fefd"} Nov 03 08:08:23.740964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.740902 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" podStartSLOduration=10.740883025 podStartE2EDuration="10.740883025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:23.737238085 +0000 UTC m=+843.118689003" watchObservedRunningTime="2026-11-03 08:08:23.740883025 +0000 UTC m=+843.122333955" Nov 03 08:08:23.755927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.755870 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" podStartSLOduration=6.755853328 podStartE2EDuration="6.755853328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:23.750839433 +0000 UTC m=+843.132290349" watchObservedRunningTime="2026-11-03 08:08:23.755853328 +0000 UTC m=+843.137304247" Nov 03 08:08:23.767911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:23.767512 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podStartSLOduration=7.767447664 podStartE2EDuration="7.767447664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:23.766747673 +0000 UTC m=+843.148198590" watchObservedRunningTime="2026-11-03 08:08:23.767447664 +0000 UTC m=+843.148898579" Nov 03 08:08:24.562796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.562759 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:24.563446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.563418 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:24.563549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.563477 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:24.577347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.577307 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q"] Nov 03 08:08:24.582368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.582346 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:24.592013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.591984 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q"] Nov 03 08:08:24.670668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.664061 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0b397f45-dockercfg-ws6nw\"" Nov 03 08:08:24.670668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.664190 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0b397f45\"" Nov 03 08:08:24.858728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.851777 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5x7\" (UniqueName: \"kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7\") pod \"payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:24.858728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:24.851873 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:25.016705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.015252 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:25.016705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.015730 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5x7\" (UniqueName: \"kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7\") pod \"payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:25.016999 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.016970 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:25.084407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.084356 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5x7\" (UniqueName: \"kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7\") pod \"payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:25.237720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.237165 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:25.563712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.563672 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:25.564858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.564788 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:25.595667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:25.594854 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q"] Nov 03 08:08:26.076319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:26.076275 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" event={"ID":"2f1c7999-a6ca-42c4-8116-cd5beb234062","Type":"ContainerStarted","Data":"f5ff7247d2569a225b2031ab14ba0558b62363b0955ae22b1739b31ceefc3126"} Nov 03 08:08:26.076508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:26.076342 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" event={"ID":"2f1c7999-a6ca-42c4-8116-cd5beb234062","Type":"ContainerStarted","Data":"b079a649f29a01f6df8b8d19acd200bb5103e6553b7c104dbb6ee19fe64fbb4b"} Nov 03 08:08:26.076508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:26.076364 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:26.093259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:26.093200 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" podStartSLOduration=2.093180509 podStartE2EDuration="2.093180509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:26.088967684 +0000 UTC m=+845.470418602" watchObservedRunningTime="2026-11-03 08:08:26.093180509 +0000 UTC m=+845.474631426" Nov 03 08:08:26.562602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:26.562566 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:26.562798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:26.562630 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:27.565704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:27.563298 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:27.565704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:27.563367 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:28.562551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:28.562516 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:28.562745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:28.562576 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:29.562580 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:29.562541 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:29.563101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:29.562601 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:30.531904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.531860 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr"] Nov 03 08:08:30.547222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.541308 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.547222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.547032 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 08:08:30.547222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.547103 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 08:08:30.547222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.547224 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vk5zp\"" Nov 03 08:08:30.547588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.547290 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 08:08:30.550034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.550009 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr"] Nov 03 08:08:30.562537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.562511 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 08:08:30.562681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.562564 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 08:08:30.684111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684067 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b91f2daf-2470-4286-928b-c3fe71832aba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.684296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684135 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91f2daf-2470-4286-928b-c3fe71832aba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.684296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684267 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkgc\" (UniqueName: \"kubernetes.io/projected/b91f2daf-2470-4286-928b-c3fe71832aba-kube-api-access-xtkgc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.684396 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684308 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.684451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684421 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91f2daf-2470-4286-928b-c3fe71832aba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.684596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684578 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.684764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.684737 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.786875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.786773 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91f2daf-2470-4286-928b-c3fe71832aba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.802293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.802254 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.802678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.802632 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.803138 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.803114 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b91f2daf-2470-4286-928b-c3fe71832aba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.827813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.827778 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91f2daf-2470-4286-928b-c3fe71832aba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.828164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.828142 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkgc\" (UniqueName: \"kubernetes.io/projected/b91f2daf-2470-4286-928b-c3fe71832aba-kube-api-access-xtkgc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.828304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.828287 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.829073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.805086 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b91f2daf-2470-4286-928b-c3fe71832aba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.842244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.842210 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.870670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.863355 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.870670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.863720 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b91f2daf-2470-4286-928b-c3fe71832aba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.870670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.863947 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91f2daf-2470-4286-928b-c3fe71832aba-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.870670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.864613 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkgc\" (UniqueName: \"kubernetes.io/projected/b91f2daf-2470-4286-928b-c3fe71832aba-kube-api-access-xtkgc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:30.882578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:30.882280 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91f2daf-2470-4286-928b-c3fe71832aba-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wmqsr\" (UID: \"b91f2daf-2470-4286-928b-c3fe71832aba\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:31.161148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:31.161067 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:31.515725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:31.515146 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr"] Nov 03 08:08:31.570312 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:31.570287 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:08:31.751078 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:31.751043 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:31.751614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:31.751596 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:31.752325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:31.752306 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:08:32.080838 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:08:32.080805 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:08:32.179532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:32.179490 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" event={"ID":"b91f2daf-2470-4286-928b-c3fe71832aba","Type":"ContainerStarted","Data":"b93aef97786db4d067fa0942d84d8f52b014357ad2416fef641fb2292e279bb0"} Nov 03 08:08:35.069469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:35.069438 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:08:36.080327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:36.080294 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:08:36.341330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:36.340693 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:36.341330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:36.340813 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:37.124540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:37.124505 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:08:37.275162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:37.275125 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" event={"ID":"b91f2daf-2470-4286-928b-c3fe71832aba","Type":"ContainerStarted","Data":"e1e7952038c27aa9760878d1c40a680954ec84053747a4222bf3f738ed4a88eb"} Nov 03 08:08:37.275337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:37.275286 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:37.277120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:37.277094 5951 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wmqsr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:08:37.277241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:37.277140 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" podUID="b91f2daf-2470-4286-928b-c3fe71832aba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:08:37.306460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:37.306397 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" podStartSLOduration=2.542865069 podStartE2EDuration="7.306378372s" totalImagesPullingTime="4.763513303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:37.296888946 +0000 UTC m=+856.678339864" watchObservedRunningTime="2026-11-03 08:08:37.306378372 +0000 UTC m=+856.687829290" Nov 03 08:08:38.296682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:38.296621 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wmqsr" Nov 03 08:08:48.643768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:48.643719 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:49.072746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.040386 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:49.204670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.200315 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:49.615743 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.537226 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:49.646166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.646123 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:49.695707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.650900 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:08:49.695707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.662730 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:49.796715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.792488 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 08:08:49.932332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.896809 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:49.951781 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.951728 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.002329 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.952252 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:50.002329 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:49.952286 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.048674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.027594 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.048674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.047430 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.095674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.060020 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:08:50.203347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.203246 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.217793 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.204094 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:08:50.555035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.544593 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.555035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.545027 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.555035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.545207 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:08:50.688419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.688369 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.720877 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.697170 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:50.742357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.721511 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:08:50.965498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.964845 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:50.965498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.965129 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.965498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.965292 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:50.989693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.973881 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:50.989693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:50.974111 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 03 08:08:51.107900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.074270 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:51.112063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.112029 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:08:51.112372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.112354 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:08:51.112526 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.112513 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:08:51.131489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.131458 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:08:51.258835 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.254402 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:51.552743 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.547738 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:51.566682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.555776 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:51.566682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.558222 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:08:51.693919 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.693873 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:51.764706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:51.731674 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.050665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.050598 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.135671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.050888 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:52.136532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.050916 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:52.149510 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.050946 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.180670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.099533 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.261396 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.260750 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:52.261396 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.261078 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.587673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.572078 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.587673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.572464 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.763194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.763129 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:52.766984 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:52.766858 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:53.013360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.013258 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:53.013833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.013803 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Nov 03 08:08:53.079367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.079327 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:53.079707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.079674 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:53.110934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.110877 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:53.126818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.126773 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:53.156941 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.156876 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:53.158401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.158349 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:53.239812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.239767 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:53.349670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.325095 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:53.349670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.305262 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:53.349670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.244245 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:53.574180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.574127 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:53.576137 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.575409 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:53.788741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.779129 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:53.788741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:53.779757 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:54.154191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.154088 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:54.212938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.189473 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:54.212938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.189812 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:54.344364 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.344317 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:54.402668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.398758 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:54.459840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.403867 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:54.578667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.577737 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:08:54.578667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.578229 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:54.767872 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.767752 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:54.768144 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.768113 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:54.814677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.812610 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:54.827673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:54.817039 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.030734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.023368 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.192962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.123877 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.193588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.164750 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.213192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.194683 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:55.215701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.196723 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:55.262687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.250930 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:55.262687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.251032 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.262687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.251722 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:55.262687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.251795 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.358182 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.358071 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.440692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.439113 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:55.440692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.439174 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.453849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.453792 5951 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:55.495218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.454140 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.495218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.482100 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.513692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.510816 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.580009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.579962 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.607730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.586721 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.793896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.793865 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:55.815678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.794135 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.826672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.816307 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:55.869358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:55.822772 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:56.132626 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.132530 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:56.132924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.132890 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.152600 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.133512 5951 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:56.152905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.152872 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.153076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.147196 5951 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:08:56.153214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.153193 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.217337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.217279 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.219167 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.219112 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:56.250934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.250118 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.358861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.358814 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.508824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.508730 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:56.526813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.522840 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:56.650480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.650437 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:56.651593 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.651093 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:08:56.934671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.863799 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:56.963359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.865448 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Nov 03 08:08:56.967721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:56.893832 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.238956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.238853 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:57.264795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.264753 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:57.267286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.266935 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.361723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.361320 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.390139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.390096 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.517008 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.516902 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.584670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.580989 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.654787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.654735 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:57.658795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.658736 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:57.861668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.861231 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded" Nov 03 08:08:57.955674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:57.955324 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:58.045061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.043938 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:08:58.286699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.281287 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:58.292568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.291122 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:58.292568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.291751 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:08:58.368893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.363757 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:08:58.536670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.535800 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:58.617859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.617747 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:58.657979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.657937 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:58.663305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.663267 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:58.983563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:58.983457 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:08:59.044792 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.044744 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:59.282824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.282784 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:08:59.313031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.311089 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:59.322791 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.311128 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:08:59.371756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.370287 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:59.540845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.540740 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:08:59.619860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.619817 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:08:59.661797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.661754 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:08:59.663086 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.663057 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59lkc" podUID="efcd3e0b-6ce9-4cbb-bf92-cad5f81b7b1c" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:08:59.670895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.670866 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:08:59.884004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.883918 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:08:59.987850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:08:59.987802 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:00.027894 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.021734 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 03 08:09:00.045834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.045772 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:09:00.320046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.320001 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:00.345804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.345754 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:09:00.384300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.384260 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:00.554954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.554849 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 08:09:00.572369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.572286 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:09:00.631191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.630458 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:09:00.942283 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.942141 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955404 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955475 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955576 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955619 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955675 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955706 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955751 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:09:00.957729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:00.955781 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:09:02.061606 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:02.061556 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:09:07.100120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:07.100081 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4"] Nov 03 08:09:07.978487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:07.978446 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq"] Nov 03 08:09:08.624684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:08.618346 5951 generic.go:356] "Generic (PLEG): container finished" podID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerID="bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f" exitCode=0 Nov 03 08:09:08.624684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:08.618657 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" event={"ID":"424a6ef4-dfba-484d-b43f-481d3b09901d","Type":"ContainerDied","Data":"bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f"} Nov 03 08:09:08.627399 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:08.625792 5951 scope.go:122] "RemoveContainer" containerID="bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f" Nov 03 08:09:08.644534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:08.629120 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:09:08.644534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:08.629212 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:09:08.644534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:08.629256 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:09:09.085210 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:09.085174 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q"] Nov 03 08:09:09.715674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:09.704869 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" event={"ID":"424a6ef4-dfba-484d-b43f-481d3b09901d","Type":"ContainerStarted","Data":"e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1"} Nov 03 08:09:10.225227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:10.225188 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:09:10.225427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:10.225373 5951 patch_prober.go:29] interesting pod/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:09:10.225505 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:10.225420 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 08:09:10.466877 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:10.466842 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:09:11.247673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:11.228031 5951 patch_prober.go:29] interesting pod/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:09:11.247673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:11.228091 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 08:09:12.155677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:12.155045 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" podUID="7ee80e0b-21ba-4435-8680-b9fbf3ed1261" containerName="payload-pre-processing" containerID="cri-o://b085533f82c0ef462232d343c3f78b8cf0f8a9917fe18d25a6fc8995e34cfb1a" gracePeriod=25 Nov 03 08:09:12.271716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:12.239775 5951 patch_prober.go:29] interesting pod/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:09:12.271716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:12.239850 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 08:09:13.174466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.094255 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" podUID="52311042-57c3-41d3-a6bc-cb01768eb10d" containerName="payload-processing" containerID="cri-o://6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c" gracePeriod=25 Nov 03 08:09:13.362107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.350913 5951 patch_prober.go:29] interesting pod/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:09:13.362107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.350990 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 08:09:13.634691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.620885 5951 generic.go:356] "Generic (PLEG): container finished" podID="7ee80e0b-21ba-4435-8680-b9fbf3ed1261" containerID="b085533f82c0ef462232d343c3f78b8cf0f8a9917fe18d25a6fc8995e34cfb1a" exitCode=0 Nov 03 08:09:13.681671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.641033 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk"] Nov 03 08:09:13.937744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.781539 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:13.945521 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.902056 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:13.982306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.982251 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" event={"ID":"7ee80e0b-21ba-4435-8680-b9fbf3ed1261","Type":"ContainerDied","Data":"b085533f82c0ef462232d343c3f78b8cf0f8a9917fe18d25a6fc8995e34cfb1a"} Nov 03 08:09:14.004668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:13.984427 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" podUID="be9d7b52-fe82-49f0-a609-1493cf8020fc" containerName="istio-proxy" containerID="cri-o://dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79" gracePeriod=30 Nov 03 08:09:14.094162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.094117 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" podUID="2f1c7999-a6ca-42c4-8116-cd5beb234062" containerName="payload-processing" containerID="cri-o://f5ff7247d2569a225b2031ab14ba0558b62363b0955ae22b1739b31ceefc3126" gracePeriod=25 Nov 03 08:09:14.157860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.157799 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:14.172702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.159231 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:14.172702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.159571 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:09:14.180979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.180950 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:09:14.237770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.229274 5951 patch_prober.go:29] interesting pod/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 08:09:14.245048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.243713 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 08:09:14.248821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.240318 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume\") pod \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " Nov 03 08:09:14.255116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.251514 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f\" (UniqueName: \"kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f\") pod \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\" (UID: \"7ee80e0b-21ba-4435-8680-b9fbf3ed1261\") " Nov 03 08:09:14.255116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.254068 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume" pod "7ee80e0b-21ba-4435-8680-b9fbf3ed1261" (UID: "7ee80e0b-21ba-4435-8680-b9fbf3ed1261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:14.255116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.255086 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:14.260202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.260162 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f" pod "7ee80e0b-21ba-4435-8680-b9fbf3ed1261" (UID: "7ee80e0b-21ba-4435-8680-b9fbf3ed1261"). InnerVolumeSpecName "kube-api-access-nvq4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:14.357393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.357361 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvq4f\" (UniqueName: \"kubernetes.io/projected/7ee80e0b-21ba-4435-8680-b9fbf3ed1261-kube-api-access-nvq4f\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:14.646105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.645067 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:09:14.649990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.649951 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm"] Nov 03 08:09:14.652668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.650189 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" containerID="cri-o://e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1" gracePeriod=30 Nov 03 08:09:14.662674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.662397 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw\" (UniqueName: \"kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw\") pod \"52311042-57c3-41d3-a6bc-cb01768eb10d\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " Nov 03 08:09:14.662674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.662474 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume\") pod \"52311042-57c3-41d3-a6bc-cb01768eb10d\" (UID: \"52311042-57c3-41d3-a6bc-cb01768eb10d\") " Nov 03 08:09:14.663159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.662952 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume" pod "52311042-57c3-41d3-a6bc-cb01768eb10d" (UID: "52311042-57c3-41d3-a6bc-cb01768eb10d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:14.663159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.663120 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52311042-57c3-41d3-a6bc-cb01768eb10d-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:14.689080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.689040 5951 generic.go:356] "Generic (PLEG): container finished" podID="2f1c7999-a6ca-42c4-8116-cd5beb234062" containerID="f5ff7247d2569a225b2031ab14ba0558b62363b0955ae22b1739b31ceefc3126" exitCode=0 Nov 03 08:09:14.689293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.689273 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" event={"ID":"2f1c7999-a6ca-42c4-8116-cd5beb234062","Type":"ContainerDied","Data":"f5ff7247d2569a225b2031ab14ba0558b62363b0955ae22b1739b31ceefc3126"} Nov 03 08:09:14.692527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.692496 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" event={"ID":"7ee80e0b-21ba-4435-8680-b9fbf3ed1261","Type":"ContainerDied","Data":"0aec5313a51c89d2796c6eb1d97cf4ba3d07e8a37f84630e8155a3b189618e17"} Nov 03 08:09:14.692734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.692563 5951 scope.go:122] "RemoveContainer" containerID="b085533f82c0ef462232d343c3f78b8cf0f8a9917fe18d25a6fc8995e34cfb1a" Nov 03 08:09:14.692828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.692791 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4" Nov 03 08:09:14.710385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.704841 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw" pod "52311042-57c3-41d3-a6bc-cb01768eb10d" (UID: "52311042-57c3-41d3-a6bc-cb01768eb10d"). InnerVolumeSpecName "kube-api-access-v2tfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:14.765067 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.765032 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tfw\" (UniqueName: \"kubernetes.io/projected/52311042-57c3-41d3-a6bc-cb01768eb10d-kube-api-access-v2tfw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:14.774508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.774176 5951 generic.go:356] "Generic (PLEG): container finished" podID="52311042-57c3-41d3-a6bc-cb01768eb10d" containerID="6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c" exitCode=0 Nov 03 08:09:14.774508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.774279 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" Nov 03 08:09:14.774679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.774555 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" event={"ID":"52311042-57c3-41d3-a6bc-cb01768eb10d","Type":"ContainerDied","Data":"6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c"} Nov 03 08:09:14.774679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.774618 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq" event={"ID":"52311042-57c3-41d3-a6bc-cb01768eb10d","Type":"ContainerDied","Data":"85c6282bad0f338948b9e3b29fa432eaf7d5f3a87d7fd72c385a282fedaec349"} Nov 03 08:09:14.776012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.775992 5951 scope.go:122] "RemoveContainer" containerID="6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c" Nov 03 08:09:14.847732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.847699 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4"] Nov 03 08:09:14.852063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.852015 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a9dd59a-7df897547-7v4r4"] Nov 03 08:09:14.988669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.910825 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:09:14.988669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:14.934525 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq"] Nov 03 08:09:15.033549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.027827 5951 scope.go:122] "RemoveContainer" containerID="6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c" Nov 03 08:09:15.045669 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:15.039814 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c\": container with ID starting with 6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c not found: ID does not exist" containerID="6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c" Nov 03 08:09:15.045669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.039858 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c"} err="failed to get container status \"6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c\": rpc error: code = NotFound desc = could not find container \"6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c\": container with ID starting with 6d6ab6ae7224c95078200a86d1b4b0ebbc299ace41752e301fa6167d92603d7c not found: ID does not exist" Nov 03 08:09:15.066909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.061196 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c855f06a-6c6587b7c5-sddmq"] Nov 03 08:09:15.169670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.169550 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7\" (UniqueName: \"kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7\") pod \"2f1c7999-a6ca-42c4-8116-cd5beb234062\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " Nov 03 08:09:15.169670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.169638 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume\") pod \"2f1c7999-a6ca-42c4-8116-cd5beb234062\" (UID: \"2f1c7999-a6ca-42c4-8116-cd5beb234062\") " Nov 03 08:09:15.194667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.190770 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume" pod "2f1c7999-a6ca-42c4-8116-cd5beb234062" (UID: "2f1c7999-a6ca-42c4-8116-cd5beb234062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:15.251909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.251831 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7" pod "2f1c7999-a6ca-42c4-8116-cd5beb234062" (UID: "2f1c7999-a6ca-42c4-8116-cd5beb234062"). InnerVolumeSpecName "kube-api-access-bp5x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:15.289125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.289064 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5x7\" (UniqueName: \"kubernetes.io/projected/2f1c7999-a6ca-42c4-8116-cd5beb234062-kube-api-access-bp5x7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:15.289125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.289096 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1c7999-a6ca-42c4-8116-cd5beb234062-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:15.999667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.996169 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" event={"ID":"2f1c7999-a6ca-42c4-8116-cd5beb234062","Type":"ContainerDied","Data":"b079a649f29a01f6df8b8d19acd200bb5103e6553b7c104dbb6ee19fe64fbb4b"} Nov 03 08:09:15.999667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.996245 5951 scope.go:122] "RemoveContainer" containerID="f5ff7247d2569a225b2031ab14ba0558b62363b0955ae22b1739b31ceefc3126" Nov 03 08:09:15.999667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:15.996377 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q" Nov 03 08:09:16.359415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:16.358238 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q"] Nov 03 08:09:16.359415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:16.358314 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0b397f45-56784b99b9-9z84q"] Nov 03 08:09:16.744552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:16.744468 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1c7999-a6ca-42c4-8116-cd5beb234062" path="/var/lib/kubelet/pods/2f1c7999-a6ca-42c4-8116-cd5beb234062/volumes" Nov 03 08:09:16.745210 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:16.745190 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52311042-57c3-41d3-a6bc-cb01768eb10d" path="/var/lib/kubelet/pods/52311042-57c3-41d3-a6bc-cb01768eb10d/volumes" Nov 03 08:09:16.745770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:16.745752 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee80e0b-21ba-4435-8680-b9fbf3ed1261" path="/var/lib/kubelet/pods/7ee80e0b-21ba-4435-8680-b9fbf3ed1261/volumes" Nov 03 08:09:21.001669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:20.958282 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:09:21.001669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:20.968049 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:09:22.000663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.000608 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:09:22.089110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089068 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089230 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089303 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089355 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089478 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089604 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089747 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089796 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.089932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.089842 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket\") pod \"be9d7b52-fe82-49f0-a609-1493cf8020fc\" (UID: \"be9d7b52-fe82-49f0-a609-1493cf8020fc\") " Nov 03 08:09:22.091415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.091386 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.091679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.091609 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.091949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.091927 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.092266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.092244 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:22.092564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.092543 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.096164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.094870 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:22.096164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.095774 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:09:22.096892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.096860 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "kube-api-access-f5jn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:22.102431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.102106 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy" pod "be9d7b52-fe82-49f0-a609-1493cf8020fc" (UID: "be9d7b52-fe82-49f0-a609-1493cf8020fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.186156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.185870 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:09:22.192279 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192198 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9d7b52-fe82-49f0-a609-1493cf8020fc-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.192491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192474 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.192591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192580 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.192713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192700 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.192791 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192780 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.192878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192868 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.192956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.192946 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9d7b52-fe82-49f0-a609-1493cf8020fc-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.193045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.193033 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jn6\" (UniqueName: \"kubernetes.io/projected/be9d7b52-fe82-49f0-a609-1493cf8020fc-kube-api-access-f5jn6\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.193125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.193114 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9d7b52-fe82-49f0-a609-1493cf8020fc-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.196127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.195875 5951 generic.go:356] "Generic (PLEG): container finished" podID="be9d7b52-fe82-49f0-a609-1493cf8020fc" containerID="dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79" exitCode=0 Nov 03 08:09:22.196127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.195950 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" event={"ID":"be9d7b52-fe82-49f0-a609-1493cf8020fc","Type":"ContainerDied","Data":"dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79"} Nov 03 08:09:22.196127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.196010 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" event={"ID":"be9d7b52-fe82-49f0-a609-1493cf8020fc","Type":"ContainerDied","Data":"f52a0dff0e61549de1832715b56469c806651c6a102bdb0ed89dc81a507ef5d4"} Nov 03 08:09:22.196127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.196039 5951 scope.go:122] "RemoveContainer" containerID="dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79" Nov 03 08:09:22.196739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.196718 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk" Nov 03 08:09:22.199978 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.199928 5951 generic.go:356] "Generic (PLEG): container finished" podID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerID="e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1" exitCode=0 Nov 03 08:09:22.200097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.200026 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" event={"ID":"424a6ef4-dfba-484d-b43f-481d3b09901d","Type":"ContainerDied","Data":"e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1"} Nov 03 08:09:22.200097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.200074 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" event={"ID":"424a6ef4-dfba-484d-b43f-481d3b09901d","Type":"ContainerDied","Data":"e36782d92dfd919c1b2809368816c26e99131c3a1c45819f3306f1d068e20556"} Nov 03 08:09:22.200223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.200150 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm" Nov 03 08:09:22.287623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.287603 5951 scope.go:122] "RemoveContainer" containerID="dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79" Nov 03 08:09:22.288084 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:22.288053 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79\": container with ID starting with dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79 not found: ID does not exist" containerID="dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79" Nov 03 08:09:22.288218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.288089 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79"} err="failed to get container status \"dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79\": rpc error: code = NotFound desc = could not find container \"dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79\": container with ID starting with dfdf01abef7b852a9f26b9cc7acc28e264175a25558293ff7b89285956a75c79 not found: ID does not exist" Nov 03 08:09:22.288218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.288107 5951 scope.go:122] "RemoveContainer" containerID="e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1" Nov 03 08:09:22.289798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.289775 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk"] Nov 03 08:09:22.294144 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294115 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294262 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294356 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294406 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294525 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294582 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294611 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294743 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.294815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.294801 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token\") pod \"424a6ef4-dfba-484d-b43f-481d3b09901d\" (UID: \"424a6ef4-dfba-484d-b43f-481d3b09901d\") " Nov 03 08:09:22.296418 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.296391 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.296608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.296590 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.296842 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.296820 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.297535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.297510 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.297905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.297878 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:22.298806 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.298783 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:22.298901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.298817 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "kube-api-access-pv5jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:22.299690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.299666 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:09:22.300911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.300886 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0a9dd59a-openshift-default-5bbd795bc5-7zfxk"] Nov 03 08:09:22.302126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.302097 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy" pod "424a6ef4-dfba-484d-b43f-481d3b09901d" (UID: "424a6ef4-dfba-484d-b43f-481d3b09901d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:09:22.303866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.303852 5951 scope.go:122] "RemoveContainer" containerID="bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f" Nov 03 08:09:22.326395 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.326279 5951 scope.go:122] "RemoveContainer" containerID="e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1" Nov 03 08:09:22.327189 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:22.327145 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1\": container with ID starting with e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1 not found: ID does not exist" containerID="e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1" Nov 03 08:09:22.327315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.327186 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1"} err="failed to get container status \"e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1\": rpc error: code = NotFound desc = could not find container \"e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1\": container with ID starting with e2d3616a46319828bc632ce4202a1362a8cd8138b8d7e642f66a8a8f125834a1 not found: ID does not exist" Nov 03 08:09:22.327315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.327214 5951 scope.go:122] "RemoveContainer" containerID="bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f" Nov 03 08:09:22.327702 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:22.327631 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f\": container with ID starting with bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f not found: ID does not exist" containerID="bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f" Nov 03 08:09:22.327854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.327830 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f"} err="failed to get container status \"bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f\": rpc error: code = NotFound desc = could not find container \"bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f\": container with ID starting with bc291d9486af36ae0699b97a9c96377dbf50bd3214329abf1e07019c36ead02f not found: ID does not exist" Nov 03 08:09:22.398227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398187 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424a6ef4-dfba-484d-b43f-481d3b09901d-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398221 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398231 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398241 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398250 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398259 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398267 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398279 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424a6ef4-dfba-484d-b43f-481d3b09901d-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.398513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.398287 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5jh\" (UniqueName: \"kubernetes.io/projected/424a6ef4-dfba-484d-b43f-481d3b09901d-kube-api-access-pv5jh\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:22.565849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.565808 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm"] Nov 03 08:09:22.575591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.575558 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c855f06a-openshift-default-978547d7c-p85lm"] Nov 03 08:09:22.934105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.934067 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" path="/var/lib/kubelet/pods/424a6ef4-dfba-484d-b43f-481d3b09901d/volumes" Nov 03 08:09:22.935231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:22.935201 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9d7b52-fe82-49f0-a609-1493cf8020fc" path="/var/lib/kubelet/pods/be9d7b52-fe82-49f0-a609-1493cf8020fc/volumes" Nov 03 08:09:27.242157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242118 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l"] Nov 03 08:09:27.242871 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242855 5951 cpu_manager.go:414] "containerMap: removing container" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242873 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242888 5951 cpu_manager.go:414] "containerMap: removing container" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242895 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242907 5951 cpu_manager.go:414] "containerMap: removing container" podUID="be9d7b52-fe82-49f0-a609-1493cf8020fc" containerName="istio-proxy" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242911 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9d7b52-fe82-49f0-a609-1493cf8020fc" containerName="istio-proxy" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242927 5951 cpu_manager.go:414] "containerMap: removing container" podUID="7ee80e0b-21ba-4435-8680-b9fbf3ed1261" containerName="payload-pre-processing" Nov 03 08:09:27.242936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242933 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee80e0b-21ba-4435-8680-b9fbf3ed1261" containerName="payload-pre-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242947 5951 cpu_manager.go:414] "containerMap: removing container" podUID="52311042-57c3-41d3-a6bc-cb01768eb10d" containerName="payload-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242952 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52311042-57c3-41d3-a6bc-cb01768eb10d" containerName="payload-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242982 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2f1c7999-a6ca-42c4-8116-cd5beb234062" containerName="payload-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.242987 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1c7999-a6ca-42c4-8116-cd5beb234062" containerName="payload-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.243132 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="52311042-57c3-41d3-a6bc-cb01768eb10d" containerName="payload-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.243163 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1c7999-a6ca-42c4-8116-cd5beb234062" containerName="payload-processing" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.243175 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" Nov 03 08:09:27.243187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.243186 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="424a6ef4-dfba-484d-b43f-481d3b09901d" containerName="istio-proxy" Nov 03 08:09:27.243493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.243197 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9d7b52-fe82-49f0-a609-1493cf8020fc" containerName="istio-proxy" Nov 03 08:09:27.243493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.243209 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee80e0b-21ba-4435-8680-b9fbf3ed1261" containerName="payload-pre-processing" Nov 03 08:09:27.249186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.249165 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.252989 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.252961 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c53cba8d-serving-cert\"" Nov 03 08:09:27.256813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.256784 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l"] Nov 03 08:09:27.341296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.341260 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.341480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.341386 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.341480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.341454 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.341564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.341549 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wzk\" (UniqueName: \"kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.443375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.443332 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.443567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.443419 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.443567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.443485 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.443567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.443552 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wzk\" (UniqueName: \"kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.444039 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.444009 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.446265 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.446239 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.446372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.446287 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.451258 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.451232 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wzk\" (UniqueName: \"kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk\") pod \"maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:27.589670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:27.589335 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:28.101973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:28.101174 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l"] Nov 03 08:09:28.116671 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:09:28.109344 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b945c3_079b_418a_a203_14cb7424b164.slice/crio-a66cd7a133b7579e89cf512bf1e3e81220ea39c4d748aec27f85955d5658a112 WatchSource:0}: Error finding container a66cd7a133b7579e89cf512bf1e3e81220ea39c4d748aec27f85955d5658a112: Status 404 returned error can't find the container with id a66cd7a133b7579e89cf512bf1e3e81220ea39c4d748aec27f85955d5658a112 Nov 03 08:09:28.293001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:28.292950 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" event={"ID":"b5b945c3-079b-418a-a203-14cb7424b164","Type":"ContainerStarted","Data":"a66cd7a133b7579e89cf512bf1e3e81220ea39c4d748aec27f85955d5658a112"} Nov 03 08:09:29.308691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:29.308385 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" event={"ID":"b5b945c3-079b-418a-a203-14cb7424b164","Type":"ContainerStarted","Data":"824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3"} Nov 03 08:09:29.309334 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:29.309306 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:29.330110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:29.330042 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" podStartSLOduration=1.767980954 podStartE2EDuration="2.330022681s" totalImagesPullingTime="562.041727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:29.327226029 +0000 UTC m=+908.708676948" watchObservedRunningTime="2026-11-03 08:09:29.330022681 +0000 UTC m=+908.711473610" Nov 03 08:09:31.228729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:31.225746 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-726m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:09:31.228729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:31.225814 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:31.794839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:31.794794 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:09:32.000059 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:31.999988 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:09:36.651721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:36.642151 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:51.025898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:51.025849 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w"] Nov 03 08:09:51.042126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:51.042086 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4771903-kkp9w"] Nov 03 08:09:52.469120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.469080 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm"] Nov 03 08:09:52.473929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.473895 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.477854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.477124 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-fe584a-dockercfg-646nd\"" Nov 03 08:09:52.477854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.477210 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-fe584a\"" Nov 03 08:09:52.483240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.483218 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm"] Nov 03 08:09:52.512277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.512238 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl"] Nov 03 08:09:52.518534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.518503 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:52.529889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.529849 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl"] Nov 03 08:09:52.551156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.551103 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slldn\" (UniqueName: \"kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn\") pod \"payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.554085 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.551702 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.768683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.768068 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:52.768683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.768253 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxn7\" (UniqueName: \"kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7\") pod \"payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:52.768683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.768320 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.768683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.768388 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slldn\" (UniqueName: \"kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn\") pod \"payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.769421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.769393 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.791736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.791699 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slldn\" (UniqueName: \"kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn\") pod \"payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.805785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.791990 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:52.805785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.799802 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669" path="/var/lib/kubelet/pods/9d1ccdb9-8fda-4ba2-9ac2-bbdb5483f669/volumes" Nov 03 08:09:52.890344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.890305 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:52.890783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.890758 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxn7\" (UniqueName: \"kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7\") pod \"payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:52.892995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.892973 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:52.915726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:52.915687 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxn7\" (UniqueName: \"kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7\") pod \"payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:53.122075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.121952 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:53.258994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.253804 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm"] Nov 03 08:09:53.435659 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:09:53.435600 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3d384d_4065_42f2_9334_42ca4377f1da.slice/crio-01f80c31fbd90ce3913c844da061668aec285b725c4f159c59205a809a1bbbd6 WatchSource:0}: Error finding container 01f80c31fbd90ce3913c844da061668aec285b725c4f159c59205a809a1bbbd6: Status 404 returned error can't find the container with id 01f80c31fbd90ce3913c844da061668aec285b725c4f159c59205a809a1bbbd6 Nov 03 08:09:53.437734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.437706 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl"] Nov 03 08:09:53.989770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.989673 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" event={"ID":"693dc940-8e5b-4d53-94d5-06b60fca8bbe","Type":"ContainerStarted","Data":"958afdbc74a8eb0f1490b18430270ca2bd7e2fd07fdf51232131e5ca9fb4ec7c"} Nov 03 08:09:53.989770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.989736 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" event={"ID":"693dc940-8e5b-4d53-94d5-06b60fca8bbe","Type":"ContainerStarted","Data":"928b5fea078c001b71ec5616b94e5ca703efdc6313218f4751971e85d48fa2c0"} Nov 03 08:09:53.990302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.989978 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:09:53.991812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.991783 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" event={"ID":"6b3d384d-4065-42f2-9334-42ca4377f1da","Type":"ContainerStarted","Data":"94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238"} Nov 03 08:09:53.991943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.991834 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:09:53.991943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:53.991853 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" event={"ID":"6b3d384d-4065-42f2-9334-42ca4377f1da","Type":"ContainerStarted","Data":"01f80c31fbd90ce3913c844da061668aec285b725c4f159c59205a809a1bbbd6"} Nov 03 08:09:54.008301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:54.008236 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" podStartSLOduration=2.00821872 podStartE2EDuration="2.00821872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:54.00454969 +0000 UTC m=+933.386000603" watchObservedRunningTime="2026-11-03 08:09:54.00821872 +0000 UTC m=+933.389669638" Nov 03 08:09:54.023570 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:54.023479 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" podStartSLOduration=2.023464396 podStartE2EDuration="2.023464396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:54.01982146 +0000 UTC m=+933.401272400" watchObservedRunningTime="2026-11-03 08:09:54.023464396 +0000 UTC m=+933.404915312" Nov 03 08:09:56.872670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:56.870877 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l"] Nov 03 08:09:56.872670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:56.871071 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" podUID="b5b945c3-079b-418a-a203-14cb7424b164" containerName="maas-api" containerID="cri-o://824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3" gracePeriod=30 Nov 03 08:09:57.555740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.543806 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:57.588531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.584628 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls\") pod \"b5b945c3-079b-418a-a203-14cb7424b164\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " Nov 03 08:09:57.588531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.584907 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs\") pod \"b5b945c3-079b-418a-a203-14cb7424b164\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " Nov 03 08:09:57.588531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.585014 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk\" (UniqueName: \"kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk\") pod \"b5b945c3-079b-418a-a203-14cb7424b164\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " Nov 03 08:09:57.588531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.585077 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca\") pod \"b5b945c3-079b-418a-a203-14cb7424b164\" (UID: \"b5b945c3-079b-418a-a203-14cb7424b164\") " Nov 03 08:09:57.589076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.589050 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca" pod "b5b945c3-079b-418a-a203-14cb7424b164" (UID: "b5b945c3-079b-418a-a203-14cb7424b164"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:57.619987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.619876 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk" pod "b5b945c3-079b-418a-a203-14cb7424b164" (UID: "b5b945c3-079b-418a-a203-14cb7424b164"). InnerVolumeSpecName "kube-api-access-t4wzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:57.619987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.619956 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls" pod "b5b945c3-079b-418a-a203-14cb7424b164" (UID: "b5b945c3-079b-418a-a203-14cb7424b164"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:09:57.636854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.636796 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs" pod "b5b945c3-079b-418a-a203-14cb7424b164" (UID: "b5b945c3-079b-418a-a203-14cb7424b164"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:09:57.690255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.690097 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:57.690255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.690191 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5b945c3-079b-418a-a203-14cb7424b164-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:57.690255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.690209 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wzk\" (UniqueName: \"kubernetes.io/projected/b5b945c3-079b-418a-a203-14cb7424b164-kube-api-access-t4wzk\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:57.690255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.690227 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b945c3-079b-418a-a203-14cb7424b164-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:09:57.890665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.887818 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6"] Nov 03 08:09:57.890665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.889047 5951 cpu_manager.go:414] "containerMap: removing container" podUID="b5b945c3-079b-418a-a203-14cb7424b164" containerName="maas-api" Nov 03 08:09:57.890665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.889068 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b945c3-079b-418a-a203-14cb7424b164" containerName="maas-api" Nov 03 08:09:57.890665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.889324 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b945c3-079b-418a-a203-14cb7424b164" containerName="maas-api" Nov 03 08:09:57.895473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.894462 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.897562 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.897524 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7a6481-openshift-default-dockercfg-wzswx\"" Nov 03 08:09:57.901018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.900991 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6"] Nov 03 08:09:57.921332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921217 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921354 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921431 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921510 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921568 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z74v\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921607 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921663 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921689 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:57.921882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:57.921752 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.023710 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.023794 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.023969 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024093 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024166 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024247 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024302 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z74v\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024343 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024375 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.024891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.024866 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.025271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.025247 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.048036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.048000 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.050450 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.050159 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.052847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.052017 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.058705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.056171 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.061567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.061538 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.065227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.065200 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z74v\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.080147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.079231 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy\") pod \"e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.080147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.079721 5951 generic.go:356] "Generic (PLEG): container finished" podID="b5b945c3-079b-418a-a203-14cb7424b164" containerID="824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3" exitCode=0 Nov 03 08:09:58.080147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.079846 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" event={"ID":"b5b945c3-079b-418a-a203-14cb7424b164","Type":"ContainerDied","Data":"824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3"} Nov 03 08:09:58.080147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.079900 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" event={"ID":"b5b945c3-079b-418a-a203-14cb7424b164","Type":"ContainerDied","Data":"a66cd7a133b7579e89cf512bf1e3e81220ea39c4d748aec27f85955d5658a112"} Nov 03 08:09:58.080147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.079930 5951 scope.go:122] "RemoveContainer" containerID="824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3" Nov 03 08:09:58.080147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.080103 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l" Nov 03 08:09:58.115869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.115770 5951 scope.go:122] "RemoveContainer" containerID="824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3" Nov 03 08:09:58.116319 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:09:58.116278 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3\": container with ID starting with 824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3 not found: ID does not exist" containerID="824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3" Nov 03 08:09:58.116421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.116335 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3"} err="failed to get container status \"824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3\": rpc error: code = NotFound desc = could not find container \"824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3\": container with ID starting with 824160c15c72c681081a19371e5b4c8c5050a94b2b58adc4f37ac8e6d1bd5cb3 not found: ID does not exist" Nov 03 08:09:58.126546 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.126466 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l"] Nov 03 08:09:58.134154 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.134116 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c53cba8d-658c7c6b8c-x8c7l"] Nov 03 08:09:58.250345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.250248 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:09:58.597361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.597330 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6"] Nov 03 08:09:58.663504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.663463 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:09:58.664145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.664118 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:09:58.664304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.664289 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:09:58.745026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:58.744999 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b945c3-079b-418a-a203-14cb7424b164" path="/var/lib/kubelet/pods/b5b945c3-079b-418a-a203-14cb7424b164/volumes" Nov 03 08:09:59.029662 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:59.028479 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf"] Nov 03 08:09:59.033003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:59.032967 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3539c6a9-s5pjf"] Nov 03 08:09:59.261938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:59.261902 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" event={"ID":"ec7c1dda-db35-485e-8a09-a43d79f743d4","Type":"ContainerStarted","Data":"7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725"} Nov 03 08:09:59.262255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:59.262234 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" event={"ID":"ec7c1dda-db35-485e-8a09-a43d79f743d4","Type":"ContainerStarted","Data":"a14be07cc37069a13a5b2e6b41fb42a28f18c4186abcaeb3181c5720edea7b0a"} Nov 03 08:09:59.288720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:09:59.286946 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podStartSLOduration=2.286927693 podStartE2EDuration="2.286927693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:59.284324613 +0000 UTC m=+938.665775544" watchObservedRunningTime="2026-11-03 08:09:59.286927693 +0000 UTC m=+938.668378611" Nov 03 08:10:00.250919 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:00.250889 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:10:00.259720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:00.259694 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:10:00.275930 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:00.275897 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:10:00.287387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:00.287358 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:10:00.743367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:00.743330 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df9a36c-3ee2-403c-bec6-9ec23c8b5acb" path="/var/lib/kubelet/pods/3df9a36c-3ee2-403c-bec6-9ec23c8b5acb/volumes" Nov 03 08:10:02.065884 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:10:02.065803 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:10:05.013550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:05.013517 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:10:05.014313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:05.013603 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:10:10.279666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.271334 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn"] Nov 03 08:10:10.279666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.277317 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn"] Nov 03 08:10:10.279666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.277427 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.316440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.316408 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-7a6481-serving-cert\"" Nov 03 08:10:10.356776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.356728 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.357119 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.357095 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf76k\" (UniqueName: \"kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.357404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.357377 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.357818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.357795 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.459705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.459627 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.459916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.459763 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf76k\" (UniqueName: \"kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.459916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.459848 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.460033 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.460013 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.460378 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.460341 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.462985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.462955 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.463173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.463153 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.467225 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.467197 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf76k\" (UniqueName: \"kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k\") pod \"maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:10.672857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:10.672770 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:11.004326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:11.004284 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn"] Nov 03 08:10:11.425676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:11.422900 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" event={"ID":"169c6de9-86c4-4ccb-9204-9e37381ea007","Type":"ContainerStarted","Data":"7395d8c895346506e3ad769f719ca0801c65a38a89153ade21e53f21b4eb46fd"} Nov 03 08:10:12.440375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:12.440337 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" event={"ID":"169c6de9-86c4-4ccb-9204-9e37381ea007","Type":"ContainerStarted","Data":"0a602e3c785c94dc1027252ddd31eb8324fe13987dd6b33f9bbb9d5e8bac05ee"} Nov 03 08:10:12.441060 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:12.441040 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:12.459248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:12.459190 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" podStartSLOduration=1.914042158 podStartE2EDuration="2.459174593s" totalImagesPullingTime="545.132435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:12.455116698 +0000 UTC m=+951.836567614" watchObservedRunningTime="2026-11-03 08:10:12.459174593 +0000 UTC m=+951.840625509" Nov 03 08:10:14.067291 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.067255 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp"] Nov 03 08:10:14.072093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.072070 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.077682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.075944 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a61a48-serving-cert\"" Nov 03 08:10:14.080868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.080837 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp"] Nov 03 08:10:14.178428 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.178389 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.178614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.178545 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.178706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.178688 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjdp\" (UniqueName: \"kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.178884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.178849 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.280211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.280175 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.280385 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:10:14.280340 5951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-serving-cert: secret "maas-api-e2e-shared-a-w1-a61a48-serving-cert" not found Nov 03 08:10:14.280442 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:10:14.280430 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls podName:68cc6679-e4d0-4740-bbac-233ed50166fa nodeName:}" failed. No retries permitted until 2026-11-03 08:10:14.78040879 +0000 UTC m=+954.161859689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls") pod "maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" (UID: "68cc6679-e4d0-4740-bbac-233ed50166fa") : secret "maas-api-e2e-shared-a-w1-a61a48-serving-cert" not found Nov 03 08:10:14.280540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.280510 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.280704 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.280611 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.280786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.280737 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjdp\" (UniqueName: \"kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.281302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.281278 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.284403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.284379 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.289013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.288984 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjdp\" (UniqueName: \"kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.795716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.794061 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:14.806435 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:14.806271 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:15.008546 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:15.008513 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:15.486083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:15.486035 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp"] Nov 03 08:10:15.486441 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:10:15.486216 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cc6679_e4d0_4740_bbac_233ed50166fa.slice/crio-7d6395185e52907957153763534c69007d0b37c5e6676a809685842810794b6e WatchSource:0}: Error finding container 7d6395185e52907957153763534c69007d0b37c5e6676a809685842810794b6e: Status 404 returned error can't find the container with id 7d6395185e52907957153763534c69007d0b37c5e6676a809685842810794b6e Nov 03 08:10:16.484275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:16.484225 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" event={"ID":"68cc6679-e4d0-4740-bbac-233ed50166fa","Type":"ContainerStarted","Data":"c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30"} Nov 03 08:10:16.484459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:16.484346 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:16.484459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:16.484370 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" event={"ID":"68cc6679-e4d0-4740-bbac-233ed50166fa","Type":"ContainerStarted","Data":"7d6395185e52907957153763534c69007d0b37c5e6676a809685842810794b6e"} Nov 03 08:10:16.504917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:16.504844 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" podStartSLOduration=2.122313048 podStartE2EDuration="2.504825767s" totalImagesPullingTime="382.512719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:16.49855027 +0000 UTC m=+955.880001189" watchObservedRunningTime="2026-11-03 08:10:16.504825767 +0000 UTC m=+955.886276687" Nov 03 08:10:18.458825 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:18.458794 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:10:22.524232 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:22.524196 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:10:27.494287 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.488488 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl"] Nov 03 08:10:27.552253 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.552213 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl"] Nov 03 08:10:27.552449 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.552383 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.680669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.672066 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb887217-serving-cert\"" Nov 03 08:10:27.709748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.694458 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.711315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.711278 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.711831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.711711 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.712145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.712124 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzz2\" (UniqueName: \"kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.831668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.827981 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.859337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.859304 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.859681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.859633 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzz2\" (UniqueName: \"kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.859949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.859931 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.860990 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:10:27.843929 5951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-serving-cert: secret "maas-api-e2e-mt-cb887217-serving-cert" not found Nov 03 08:10:27.861219 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:10:27.861206 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls podName:5d3a2646-85c0-4f88-872e-6e2775a8fc32 nodeName:}" failed. No retries permitted until 2026-11-03 08:10:28.361166539 +0000 UTC m=+967.742617450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls") pod "maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" (UID: "5d3a2646-85c0-4f88-872e-6e2775a8fc32") : secret "maas-api-e2e-mt-cb887217-serving-cert" not found Nov 03 08:10:27.900371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.881319 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.900696 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.898043 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:27.903957 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:27.901351 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzz2\" (UniqueName: \"kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:28.384139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:28.384095 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:28.389576 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:28.389541 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-ps5hl\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:28.473466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:28.473427 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:28.786467 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:10:28.786425 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3a2646_85c0_4f88_872e_6e2775a8fc32.slice/crio-39da626302ef0303b012224e67898f3062d5ac0faf0faccf5a4b06c2f783f05c WatchSource:0}: Error finding container 39da626302ef0303b012224e67898f3062d5ac0faf0faccf5a4b06c2f783f05c: Status 404 returned error can't find the container with id 39da626302ef0303b012224e67898f3062d5ac0faf0faccf5a4b06c2f783f05c Nov 03 08:10:28.787538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:28.787511 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl"] Nov 03 08:10:29.807364 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.807323 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" event={"ID":"5d3a2646-85c0-4f88-872e-6e2775a8fc32","Type":"ContainerStarted","Data":"2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69"} Nov 03 08:10:29.807815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.807388 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" event={"ID":"5d3a2646-85c0-4f88-872e-6e2775a8fc32","Type":"ContainerStarted","Data":"39da626302ef0303b012224e67898f3062d5ac0faf0faccf5a4b06c2f783f05c"} Nov 03 08:10:29.807815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.807428 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:29.829468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.829412 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" podStartSLOduration=2.361450958 podStartE2EDuration="2.829397132s" totalImagesPullingTime="467.946174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:29.823577241 +0000 UTC m=+969.205028157" watchObservedRunningTime="2026-11-03 08:10:29.829397132 +0000 UTC m=+969.210848048" Nov 03 08:10:29.906040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.892507 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc"] Nov 03 08:10:29.930018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.929988 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:29.971666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:29.944133 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d6a8bf-openshift-default-dockercfg-xf5x5\"" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.048573 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.048693 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.048826 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.048989 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.049043 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.049130 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.049163 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggx5f\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.049205 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.049882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.049243 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.063619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.057632 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc"] Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.160886 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.160946 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.161046 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.161193 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.161240 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.169027 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.171597 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.171678 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.171724 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggx5f\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.171777 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.171821 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.173307 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.174049 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.174765 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.178450 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.197861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.182241 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.223620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.221389 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggx5f\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.223620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.223413 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token\") pod \"e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.473370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.473262 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:30.827001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.826961 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc"] Nov 03 08:10:30.836578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.836532 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:10:30.836721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.836660 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:10:30.836721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:30.836710 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:10:31.831066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:31.831026 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" event={"ID":"3c6aa698-50f9-4827-bfb2-3af5344613cd","Type":"ContainerStarted","Data":"3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad"} Nov 03 08:10:31.831567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:31.831096 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" event={"ID":"3c6aa698-50f9-4827-bfb2-3af5344613cd","Type":"ContainerStarted","Data":"ae8f2c3c6ac46cc5c1b328b9ccc44691ba306dcd0077119241d20c4adb7a4a66"} Nov 03 08:10:31.855579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:31.855528 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podStartSLOduration=2.855514864 podStartE2EDuration="2.855514864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:31.85000327 +0000 UTC m=+971.231454199" watchObservedRunningTime="2026-11-03 08:10:31.855514864 +0000 UTC m=+971.236965781" Nov 03 08:10:31.958223 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:10:31.958180 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:10:32.474239 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:32.474188 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:32.480244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:32.480216 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:32.844611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:32.844579 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:32.845974 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:32.845950 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:10:35.530555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.530519 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx"] Nov 03 08:10:35.535134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.535111 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.543080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.543048 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d6a8bf\"" Nov 03 08:10:35.544074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.544014 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d6a8bf-dockercfg-v78ql\"" Nov 03 08:10:35.549702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.549680 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx"] Nov 03 08:10:35.637520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.637480 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.637732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.637612 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94sw\" (UniqueName: \"kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw\") pod \"payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.739170 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.739131 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.739377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.739206 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94sw\" (UniqueName: \"kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw\") pod \"payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.739951 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.739929 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.746564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.746537 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94sw\" (UniqueName: \"kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw\") pod \"payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:35.862470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.860989 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:10:35.862470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:35.862178 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:36.508917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:36.508887 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx"] Nov 03 08:10:36.511775 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:10:36.511736 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1030fc0d_5e30_47a0_bd3e_9640d42e781c.slice/crio-005e00f60920395b22c5c4ecc37bbbb4a8e3a7c3c799ab3b27abaddea7dae6b3 WatchSource:0}: Error finding container 005e00f60920395b22c5c4ecc37bbbb4a8e3a7c3c799ab3b27abaddea7dae6b3: Status 404 returned error can't find the container with id 005e00f60920395b22c5c4ecc37bbbb4a8e3a7c3c799ab3b27abaddea7dae6b3 Nov 03 08:10:36.967185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:36.967083 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" event={"ID":"1030fc0d-5e30-47a0-bd3e-9640d42e781c","Type":"ContainerStarted","Data":"a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1"} Nov 03 08:10:36.967185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:36.967144 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" event={"ID":"1030fc0d-5e30-47a0-bd3e-9640d42e781c","Type":"ContainerStarted","Data":"005e00f60920395b22c5c4ecc37bbbb4a8e3a7c3c799ab3b27abaddea7dae6b3"} Nov 03 08:10:36.967185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:36.967173 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:36.983998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:36.983945 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" podStartSLOduration=1.983930097 podStartE2EDuration="1.983930097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:36.982163408 +0000 UTC m=+976.363614337" watchObservedRunningTime="2026-11-03 08:10:36.983930097 +0000 UTC m=+976.365381013" Nov 03 08:10:45.756185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.750290 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps"] Nov 03 08:10:45.757229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.757204 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:45.775052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.768322 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps"] Nov 03 08:10:45.798241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.798118 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-d6a8bf-serving-cert\"" Nov 03 08:10:45.901746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.901704 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lqch\" (UniqueName: \"kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:45.901998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.901968 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:45.902054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.902024 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:45.902090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:45.902078 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.005386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.004310 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lqch\" (UniqueName: \"kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.005386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.004519 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.005386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.004558 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.005386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.004587 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.006039 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.005458 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.008493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.008422 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.011543 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.011514 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.014458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.014223 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lqch\" (UniqueName: \"kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch\") pod \"maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.106846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.106806 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:46.454624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:46.454589 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps"] Nov 03 08:10:46.457192 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:10:46.457155 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a01dbb7_a4c7_4393_9da1_72e7930db782.slice/crio-122cc9175deac2466ac2b349d5f6b1be88e7d7bc86915cd8dac291e03d517436 WatchSource:0}: Error finding container 122cc9175deac2466ac2b349d5f6b1be88e7d7bc86915cd8dac291e03d517436: Status 404 returned error can't find the container with id 122cc9175deac2466ac2b349d5f6b1be88e7d7bc86915cd8dac291e03d517436 Nov 03 08:10:47.108979 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:47.108935 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" event={"ID":"1a01dbb7-a4c7-4393-9da1-72e7930db782","Type":"ContainerStarted","Data":"713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1"} Nov 03 08:10:47.109457 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:47.109010 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:47.109842 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:47.109793 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" event={"ID":"1a01dbb7-a4c7-4393-9da1-72e7930db782","Type":"ContainerStarted","Data":"122cc9175deac2466ac2b349d5f6b1be88e7d7bc86915cd8dac291e03d517436"} Nov 03 08:10:47.126056 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:47.126000 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" podStartSLOduration=1.6624204040000001 podStartE2EDuration="2.125984559s" totalImagesPullingTime="463.564155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:47.124245232 +0000 UTC m=+986.505696148" watchObservedRunningTime="2026-11-03 08:10:47.125984559 +0000 UTC m=+986.507435475" Nov 03 08:10:47.979981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:47.979953 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:10:54.149888 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:54.149856 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:10:57.230671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.229753 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.230671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.230179 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:10:57.255672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.246745 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.255672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.247240 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.255672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.247520 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:10:57.275674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.264761 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.399669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.399364 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.579262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.579216 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:10:57.695670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.693322 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.770671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.769788 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:57.801865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:57.801802 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:58.235666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.230925 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:58.235666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.231343 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:10:58.235666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.231928 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:10:58.235666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.231987 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:10:58.255683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.250736 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:10:58.255683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.251112 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:10:58.255683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.251734 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:10:58.255683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.251795 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:10:58.272693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.262191 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:10:58.272693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.262790 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:10:58.272693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.265992 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:58.272693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.266558 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:10:58.631669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.606405 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:58.656670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.613749 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:10:58.687794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.655918 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:10:58.886724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.812729 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:10:58.886724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.846295 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 08:10:58.960799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.956383 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:10:58.971955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.971924 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:10:58.972225 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:58.972211 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:10:59.234284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.234173 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:59.235913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.235867 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:10:59.257479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.257423 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:59.257767 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.257675 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:10:59.264414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.264366 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:59.392113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.386312 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:10:59.572047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.570676 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:10:59.644666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.625130 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:10:59.654005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.653960 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:59.675765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.675665 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:10:59.675765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.675730 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:10:59.692288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.691719 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:10:59.692288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.691912 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:10:59.723681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.718504 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:10:59.745669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.743980 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:10:59.748884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.748053 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:10:59.763331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:10:59.762507 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:11:02.034700 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:11:02.034636 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:11:08.063668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:08.052135 5951 generic.go:356] "Generic (PLEG): container finished" podID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerID="7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725" exitCode=0 Nov 03 08:11:08.063668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:08.052402 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" event={"ID":"ec7c1dda-db35-485e-8a09-a43d79f743d4","Type":"ContainerDied","Data":"7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725"} Nov 03 08:11:08.063668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:08.053012 5951 scope.go:122] "RemoveContainer" containerID="7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725" Nov 03 08:11:08.068664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:08.066980 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:11:08.068664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:08.067075 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:11:08.068664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:08.067127 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:11:09.063835 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:09.063799 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" event={"ID":"ec7c1dda-db35-485e-8a09-a43d79f743d4","Type":"ContainerStarted","Data":"f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67"} Nov 03 08:11:09.251017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:09.250980 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:11:09.251198 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:09.251179 5951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 08:11:09.251251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:09.251230 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 08:11:10.251321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:10.251282 5951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 08:11:10.251766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:10.251341 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 08:11:11.251514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:11.251478 5951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 08:11:11.251958 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:11.251545 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 08:11:12.251479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:12.251443 5951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 08:11:12.251721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:12.251514 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 08:11:13.252712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:13.252560 5951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 08:11:13.252712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:13.252620 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 08:11:14.252007 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:14.251977 5951 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 08:11:14.252255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:14.252231 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 08:11:15.280666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:15.278790 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:11:15.282790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:15.282760 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:11:16.215158 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:16.215129 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:11:23.145179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:23.145147 5951 scope.go:122] "RemoveContainer" containerID="596f38a4634bc585167824ea1f4ba912d8d60f45d34220fa215a356da337019d" Nov 03 08:11:23.157220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:23.157192 5951 scope.go:122] "RemoveContainer" containerID="2c5ea55d528a6a543aa24cfe9061a3f5eb6b00c0a6845cf1ab38be0d599d4a37" Nov 03 08:11:31.575971 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.575935 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8"] Nov 03 08:11:31.582892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.582866 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.593943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.593911 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2a44ba-serving-cert\"" Nov 03 08:11:31.602821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.602787 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8"] Nov 03 08:11:31.631467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.631306 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.631467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.631341 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.631467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.631443 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9mk\" (UniqueName: \"kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.631852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.631494 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.743709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.734167 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.743709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.734211 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.743709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.734281 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9mk\" (UniqueName: \"kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.743709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.734315 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.743709 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:11:31.734613 5951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-serving-cert: secret "maas-api-e2e-api-a-2a44ba-serving-cert" not found Nov 03 08:11:31.751753 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:11:31.747449 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls podName:690f04a1-abec-447c-9338-e65c4eb34a4d nodeName:}" failed. No retries permitted until 2026-11-03 08:11:32.247414069 +0000 UTC m=+1031.628864984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls") pod "maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" (UID: "690f04a1-abec-447c-9338-e65c4eb34a4d") : secret "maas-api-e2e-api-a-2a44ba-serving-cert" not found Nov 03 08:11:31.751753 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.748839 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.751753 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.750369 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.780707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:31.779496 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9mk\" (UniqueName: \"kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:31.958162 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:11:31.958061 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:11:32.349350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:32.349304 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:32.364670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:32.356403 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\") pod \"maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:32.534666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:32.527974 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:32.990719 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:32.989576 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8"] Nov 03 08:11:33.458357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:33.458309 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" event={"ID":"690f04a1-abec-447c-9338-e65c4eb34a4d","Type":"ContainerStarted","Data":"b8453c9bd2b64da64efd2df909ddd880280b7be70e41f8fb47c4048930a87359"} Nov 03 08:11:34.469439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:34.469395 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" event={"ID":"690f04a1-abec-447c-9338-e65c4eb34a4d","Type":"ContainerStarted","Data":"9569c49d1d3eefa8e2c96899024898e34cffb8bf18de590bf9ac21e381997972"} Nov 03 08:11:34.469879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:34.469456 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:34.486783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:34.486730 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" podStartSLOduration=3.017492412 podStartE2EDuration="3.486717395s" totalImagesPullingTime="469.224983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:34.483978887 +0000 UTC m=+1033.865429803" watchObservedRunningTime="2026-11-03 08:11:34.486717395 +0000 UTC m=+1033.868168312" Nov 03 08:11:40.483466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:40.483429 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:11:52.561122 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.561086 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n"] Nov 03 08:11:52.566824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.566797 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:52.569760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.569736 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e255cbee\"" Nov 03 08:11:52.569961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.569943 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e255cbee-dockercfg-pbxsq\"" Nov 03 08:11:52.582012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.581970 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:52.582195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.582162 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n"] Nov 03 08:11:52.583389 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.583355 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqr6c\" (UniqueName: \"kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c\") pod \"payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:52.818622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.818521 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqr6c\" (UniqueName: \"kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c\") pod \"payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:52.819214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.819182 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:52.822749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.822725 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:52.888676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:52.860604 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqr6c\" (UniqueName: \"kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c\") pod \"payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:53.063380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:53.063346 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:53.390441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:53.390408 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n"] Nov 03 08:11:53.393635 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:11:53.393601 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cccd504_3dc2_4c0b_af6b_8e08cb5a180e.slice/crio-4f8992562abd882feb141b1c036374101e404dadab62a0226498dff047e68b5f WatchSource:0}: Error finding container 4f8992562abd882feb141b1c036374101e404dadab62a0226498dff047e68b5f: Status 404 returned error can't find the container with id 4f8992562abd882feb141b1c036374101e404dadab62a0226498dff047e68b5f Nov 03 08:11:54.091983 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:54.091933 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" event={"ID":"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e","Type":"ContainerStarted","Data":"35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756"} Nov 03 08:11:54.092371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:54.092024 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" event={"ID":"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e","Type":"ContainerStarted","Data":"4f8992562abd882feb141b1c036374101e404dadab62a0226498dff047e68b5f"} Nov 03 08:11:54.092371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:54.092121 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:11:54.111032 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:11:54.110908 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" podStartSLOduration=2.110885857 podStartE2EDuration="2.110885857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:54.105746895 +0000 UTC m=+1053.487197811" watchObservedRunningTime="2026-11-03 08:11:54.110885857 +0000 UTC m=+1053.492336779" Nov 03 08:12:01.957314 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:12:01.957277 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:12:05.123144 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:05.123111 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:12:06.581914 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.581880 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7"] Nov 03 08:12:06.588105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.588078 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.593913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.593887 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e255cbee-serving-cert\"" Nov 03 08:12:06.610343 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.610312 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7"] Nov 03 08:12:06.776831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.776799 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.777159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.777136 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.777285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.777269 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzld8\" (UniqueName: \"kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.777577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.777550 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.879503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.879399 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.879694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.879617 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.879802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.879747 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.879878 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.879802 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzld8\" (UniqueName: \"kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.881063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.881020 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.882924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.882904 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.883332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.883312 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.893409 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.893382 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzld8\" (UniqueName: \"kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8\") pod \"maas-api-e2e-mt-e255cbee-d9d75984-2cfv7\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:06.917702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:06.917613 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:07.345679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.345078 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7"] Nov 03 08:12:07.520304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.520269 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc"] Nov 03 08:12:07.527480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.527451 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.545766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.545693 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-178e68-serving-cert\"" Nov 03 08:12:07.548667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.548617 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc"] Nov 03 08:12:07.651351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.651316 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.651864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.651666 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rttx\" (UniqueName: \"kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.651864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.651719 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.651864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.651809 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.760686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.760340 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rttx\" (UniqueName: \"kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.760686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.760416 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.760686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.760451 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.760686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.760689 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.767678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.762261 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.792706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.792603 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.792942 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.792795 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rttx\" (UniqueName: \"kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.796176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.796105 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:07.867001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:07.866966 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:08.259068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:08.235355 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc"] Nov 03 08:12:08.389192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:08.389158 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" event={"ID":"a64b095c-8435-44d7-8ca8-dab13c1f7aff","Type":"ContainerStarted","Data":"f17adda0a27d81db3af8442dc50efa722cb92206c93e0db43be3df9bab901173"} Nov 03 08:12:08.391045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:08.391013 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" event={"ID":"0c6f0149-90e4-4644-8f93-1a5f38caff96","Type":"ContainerStarted","Data":"0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f"} Nov 03 08:12:08.391186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:08.391067 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" event={"ID":"0c6f0149-90e4-4644-8f93-1a5f38caff96","Type":"ContainerStarted","Data":"74a05dde1fc98a409e773c327b150cf8dfb739c06fdfdf117924f24922fee4aa"} Nov 03 08:12:08.391186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:08.391121 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:08.419196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:08.419121 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" podStartSLOduration=1.928427841 podStartE2EDuration="2.419099426s" totalImagesPullingTime="490.671585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:08.413033079 +0000 UTC m=+1067.794483990" watchObservedRunningTime="2026-11-03 08:12:08.419099426 +0000 UTC m=+1067.800550344" Nov 03 08:12:09.404308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:09.404273 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" event={"ID":"a64b095c-8435-44d7-8ca8-dab13c1f7aff","Type":"ContainerStarted","Data":"df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e"} Nov 03 08:12:09.404862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:09.404370 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:09.426390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:09.426328 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" podStartSLOduration=1.998785975 podStartE2EDuration="2.42631198s" totalImagesPullingTime="427.526005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:09.420378379 +0000 UTC m=+1068.801829296" watchObservedRunningTime="2026-11-03 08:12:09.42631198 +0000 UTC m=+1068.807762896" Nov 03 08:12:10.186946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.186907 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh"] Nov 03 08:12:10.194149 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.194115 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.206424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.206367 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh"] Nov 03 08:12:10.215405 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.214819 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dfb883-serving-cert\"" Nov 03 08:12:10.336028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.329506 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.336028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.329675 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.336028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.329729 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.336028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.331714 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcs7\" (UniqueName: \"kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.433952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.433510 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcs7\" (UniqueName: \"kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.434719 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.434690 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.434971 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.434953 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.438330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.438247 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.440102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.440078 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.440409 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:12:10.440391 5951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-serving-cert: secret "maas-api-e2e-api-b-dfb883-serving-cert" not found Nov 03 08:12:10.440616 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:12:10.440590 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls podName:96fce170-8828-446a-ba55-b3d2b97864da nodeName:}" failed. No retries permitted until 2026-11-03 08:12:10.9405668 +0000 UTC m=+1070.322017713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls") pod "maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" (UID: "96fce170-8828-446a-ba55-b3d2b97864da") : secret "maas-api-e2e-api-b-dfb883-serving-cert" not found Nov 03 08:12:10.449152 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.448589 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.450573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.450553 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcs7\" (UniqueName: \"kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.971706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.965970 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:10.978906 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:10.978869 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\") pod \"maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:11.145422 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:11.145385 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:11.518973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:11.517039 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh"] Nov 03 08:12:12.449722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:12.449683 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" event={"ID":"96fce170-8828-446a-ba55-b3d2b97864da","Type":"ContainerStarted","Data":"f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83"} Nov 03 08:12:12.450036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:12.450006 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" event={"ID":"96fce170-8828-446a-ba55-b3d2b97864da","Type":"ContainerStarted","Data":"76910e0d44ea1403116feeeb413f388316e0a2768f69fe8372c720c8618b6bb2"} Nov 03 08:12:12.450246 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:12.450225 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:12.476810 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:12.476739 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" podStartSLOduration=2.012893753 podStartE2EDuration="2.476718909s" totalImagesPullingTime="463.825156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:12.474797653 +0000 UTC m=+1071.856248571" watchObservedRunningTime="2026-11-03 08:12:12.476718909 +0000 UTC m=+1071.858169827" Nov 03 08:12:14.411278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:14.411246 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:12:15.439496 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:15.439458 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:12:18.485356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:12:18.485318 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:12:32.080729 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:12:32.076957 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:13:01.045309 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:01.045260 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9"] Nov 03 08:13:01.049802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:01.049773 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89f6363-xnqb9"] Nov 03 08:13:01.986720 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:13:01.986623 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:13:02.742431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:02.742391 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aae1003-32cf-4467-8151-4fa91cc611af" path="/var/lib/kubelet/pods/7aae1003-32cf-4467-8151-4fa91cc611af/volumes" Nov 03 08:13:08.804214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:08.804168 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl"] Nov 03 08:13:08.804855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:08.804427 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" podUID="5d3a2646-85c0-4f88-872e-6e2775a8fc32" containerName="maas-api" containerID="cri-o://2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69" gracePeriod=30 Nov 03 08:13:09.545867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.545839 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:13:09.639050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.639010 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2\" (UniqueName: \"kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2\") pod \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " Nov 03 08:13:09.639247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.639080 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\") pod \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " Nov 03 08:13:09.641946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.641914 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs\") pod \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " Nov 03 08:13:09.642402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.642239 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca\") pod \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\" (UID: \"5d3a2646-85c0-4f88-872e-6e2775a8fc32\") " Nov 03 08:13:09.652483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.652439 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca" pod "5d3a2646-85c0-4f88-872e-6e2775a8fc32" (UID: "5d3a2646-85c0-4f88-872e-6e2775a8fc32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:13:09.656050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.656017 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2" pod "5d3a2646-85c0-4f88-872e-6e2775a8fc32" (UID: "5d3a2646-85c0-4f88-872e-6e2775a8fc32"). InnerVolumeSpecName "kube-api-access-4xzz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:09.656799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.656769 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs" pod "5d3a2646-85c0-4f88-872e-6e2775a8fc32" (UID: "5d3a2646-85c0-4f88-872e-6e2775a8fc32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:13:09.716884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.710048 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls" pod "5d3a2646-85c0-4f88-872e-6e2775a8fc32" (UID: "5d3a2646-85c0-4f88-872e-6e2775a8fc32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.734128 5951 generic.go:356] "Generic (PLEG): container finished" podID="5d3a2646-85c0-4f88-872e-6e2775a8fc32" containerID="2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69" exitCode=0 Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.734245 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" event={"ID":"5d3a2646-85c0-4f88-872e-6e2775a8fc32","Type":"ContainerDied","Data":"2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69"} Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.734294 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" event={"ID":"5d3a2646-85c0-4f88-872e-6e2775a8fc32","Type":"ContainerDied","Data":"39da626302ef0303b012224e67898f3062d5ac0faf0faccf5a4b06c2f783f05c"} Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.734326 5951 scope.go:122] "RemoveContainer" containerID="2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69" Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.734545 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl" Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.750271 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xzz2\" (UniqueName: \"kubernetes.io/projected/5d3a2646-85c0-4f88-872e-6e2775a8fc32-kube-api-access-4xzz2\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.750302 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.750317 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a2646-85c0-4f88-872e-6e2775a8fc32-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:09.750691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.750331 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a2646-85c0-4f88-872e-6e2775a8fc32-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:09.762678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.762657 5951 scope.go:122] "RemoveContainer" containerID="2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69" Nov 03 08:13:09.764681 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:13:09.764630 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69\": container with ID starting with 2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69 not found: ID does not exist" containerID="2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69" Nov 03 08:13:09.764860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.764836 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69"} err="failed to get container status \"2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69\": rpc error: code = NotFound desc = could not find container \"2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69\": container with ID starting with 2dacbc31ce47a17a1b6cab5532c4de30e3ebbd0765a10bbf3092ca50d2d7ef69 not found: ID does not exist" Nov 03 08:13:09.781782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.781737 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl"] Nov 03 08:13:09.786228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:09.786203 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-ps5hl"] Nov 03 08:13:10.754230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:10.754182 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3a2646-85c0-4f88-872e-6e2775a8fc32" path="/var/lib/kubelet/pods/5d3a2646-85c0-4f88-872e-6e2775a8fc32/volumes" Nov 03 08:13:16.764199 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:16.764158 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5"] Nov 03 08:13:16.764799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:16.764396 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" containerID="cri-o://f7806c0114ac1f2913816750d0c8d35e7ae156af88e19b5dfa67ad48e89d0c22" gracePeriod=30 Nov 03 08:13:16.769922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:16.769895 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5"] Nov 03 08:13:16.770128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:16.770105 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" containerID="cri-o://aa9656116cb1d99b3502145e020939c547c95ba18b1315bd815adea2912148ba" gracePeriod=30 Nov 03 08:13:16.777535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:16.777509 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz"] Nov 03 08:13:16.777754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:16.777731 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" containerID="cri-o://696fcc7549ed0b41be506258c68b79e1001ccf699ce10d80e347f2a538bd3c30" gracePeriod=30 Nov 03 08:13:17.410231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.410198 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62"] Nov 03 08:13:17.411145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.411121 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5d3a2646-85c0-4f88-872e-6e2775a8fc32" containerName="maas-api" Nov 03 08:13:17.411145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.411142 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3a2646-85c0-4f88-872e-6e2775a8fc32" containerName="maas-api" Nov 03 08:13:17.411351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.411295 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3a2646-85c0-4f88-872e-6e2775a8fc32" containerName="maas-api" Nov 03 08:13:17.416354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.416329 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.429022 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.428992 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9"] Nov 03 08:13:17.433500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.433473 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb887217-dockercfg-k89nk\"" Nov 03 08:13:17.436073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.436036 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62"] Nov 03 08:13:17.436300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.436160 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.436417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.436369 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.436663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.436579 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9j8\" (UniqueName: \"kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.514712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.514682 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb887217\"" Nov 03 08:13:17.618687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.595786 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9j8\" (UniqueName: \"kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.618687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.596139 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrwt\" (UniqueName: \"kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.618687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.596291 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.618687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.596356 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.618687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.597277 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.686796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.686709 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9"] Nov 03 08:13:17.709669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.709072 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.709669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.709511 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrwt\" (UniqueName: \"kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.757689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.753274 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.882724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.882688 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrwt\" (UniqueName: \"kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt\") pod \"payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:17.958813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.950565 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9j8\" (UniqueName: \"kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8\") pod \"payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:17.958813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:17.953476 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:18.048020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:18.047978 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:18.320368 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:13:18.320325 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aab0f91_f6ac_4938_a5bb_273bddbe8346.slice/crio-14a8f3a404d08f2be8b41ff49ffd60eccfbeec02efab21f3897ca61c1cf54c2a WatchSource:0}: Error finding container 14a8f3a404d08f2be8b41ff49ffd60eccfbeec02efab21f3897ca61c1cf54c2a: Status 404 returned error can't find the container with id 14a8f3a404d08f2be8b41ff49ffd60eccfbeec02efab21f3897ca61c1cf54c2a Nov 03 08:13:18.320873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:18.320851 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9"] Nov 03 08:13:18.436985 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:13:18.436937 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60394526_fb1e_4d6f_8321_f89db0abcc29.slice/crio-c284521d95955c86ccb68477784b47fbf0365ef7d3503a7e8d1c78bd16e9bb8c WatchSource:0}: Error finding container c284521d95955c86ccb68477784b47fbf0365ef7d3503a7e8d1c78bd16e9bb8c: Status 404 returned error can't find the container with id c284521d95955c86ccb68477784b47fbf0365ef7d3503a7e8d1c78bd16e9bb8c Nov 03 08:13:18.439693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:18.439618 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62"] Nov 03 08:13:19.055405 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.055361 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" event={"ID":"8aab0f91-f6ac-4938-a5bb-273bddbe8346","Type":"ContainerStarted","Data":"da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca"} Nov 03 08:13:19.055897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.055433 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" event={"ID":"8aab0f91-f6ac-4938-a5bb-273bddbe8346","Type":"ContainerStarted","Data":"14a8f3a404d08f2be8b41ff49ffd60eccfbeec02efab21f3897ca61c1cf54c2a"} Nov 03 08:13:19.056201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.056181 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:19.058156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.058018 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" event={"ID":"60394526-fb1e-4d6f-8321-f89db0abcc29","Type":"ContainerStarted","Data":"990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e"} Nov 03 08:13:19.058156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.058076 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" event={"ID":"60394526-fb1e-4d6f-8321-f89db0abcc29","Type":"ContainerStarted","Data":"c284521d95955c86ccb68477784b47fbf0365ef7d3503a7e8d1c78bd16e9bb8c"} Nov 03 08:13:19.058563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.058540 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:19.094850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.094785 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" podStartSLOduration=2.094764086 podStartE2EDuration="2.094764086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:19.091501663 +0000 UTC m=+1138.472952580" watchObservedRunningTime="2026-11-03 08:13:19.094764086 +0000 UTC m=+1138.476215004" Nov 03 08:13:19.121277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:19.121215 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" podStartSLOduration=2.121200013 podStartE2EDuration="2.121200013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:19.118815781 +0000 UTC m=+1138.500266697" watchObservedRunningTime="2026-11-03 08:13:19.121200013 +0000 UTC m=+1138.502650931" Nov 03 08:13:20.000557 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.000515 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp"] Nov 03 08:13:20.007041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.007013 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.009510 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.009474 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvf6l\" (UniqueName: \"kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l\") pod \"payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.009741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.009719 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.021564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.021539 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-94e3cb\"" Nov 03 08:13:20.029193 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.029155 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp"] Nov 03 08:13:20.111392 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.111345 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvf6l\" (UniqueName: \"kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l\") pod \"payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.111949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.111561 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.113527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.113500 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.121989 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.121963 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvf6l\" (UniqueName: \"kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l\") pod \"payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.334947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.331955 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:20.895332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:20.895289 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp"] Nov 03 08:13:20.899461 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:13:20.899424 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158efa57_2705_48b6_a778_f7afecc9fde5.slice/crio-c37a8e4f3b8426f7a966f8406b12a64489689234de036dec0fea011bdd7a21a4 WatchSource:0}: Error finding container c37a8e4f3b8426f7a966f8406b12a64489689234de036dec0fea011bdd7a21a4: Status 404 returned error can't find the container with id c37a8e4f3b8426f7a966f8406b12a64489689234de036dec0fea011bdd7a21a4 Nov 03 08:13:21.085734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:21.085686 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" event={"ID":"158efa57-2705-48b6-a778-f7afecc9fde5","Type":"ContainerStarted","Data":"8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407"} Nov 03 08:13:21.085937 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:21.085757 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" event={"ID":"158efa57-2705-48b6-a778-f7afecc9fde5","Type":"ContainerStarted","Data":"c37a8e4f3b8426f7a966f8406b12a64489689234de036dec0fea011bdd7a21a4"} Nov 03 08:13:21.085994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:21.085958 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:21.124756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:21.124626 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" podStartSLOduration=2.124608646 podStartE2EDuration="2.124608646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:21.117290917 +0000 UTC m=+1140.498741848" watchObservedRunningTime="2026-11-03 08:13:21.124608646 +0000 UTC m=+1140.506059562" Nov 03 08:13:22.104620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.101998 5951 generic.go:356] "Generic (PLEG): container finished" podID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerID="aa9656116cb1d99b3502145e020939c547c95ba18b1315bd815adea2912148ba" exitCode=0 Nov 03 08:13:22.104620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.102423 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" event={"ID":"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf","Type":"ContainerDied","Data":"aa9656116cb1d99b3502145e020939c547c95ba18b1315bd815adea2912148ba"} Nov 03 08:13:22.104620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.102480 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" event={"ID":"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf","Type":"ContainerDied","Data":"7d7200c62d9248ed9b19a851580f0b309c174d9b8751e56a6b04295718bbecee"} Nov 03 08:13:22.104620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.102519 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7200c62d9248ed9b19a851580f0b309c174d9b8751e56a6b04295718bbecee" Nov 03 08:13:22.105423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.105363 5951 generic.go:356] "Generic (PLEG): container finished" podID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerID="f7806c0114ac1f2913816750d0c8d35e7ae156af88e19b5dfa67ad48e89d0c22" exitCode=0 Nov 03 08:13:22.105560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.105462 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" event={"ID":"5ac2b7e5-0558-40a5-953c-1b3f57ce8918","Type":"ContainerDied","Data":"f7806c0114ac1f2913816750d0c8d35e7ae156af88e19b5dfa67ad48e89d0c22"} Nov 03 08:13:22.109330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.108373 5951 generic.go:356] "Generic (PLEG): container finished" podID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerID="696fcc7549ed0b41be506258c68b79e1001ccf699ce10d80e347f2a538bd3c30" exitCode=0 Nov 03 08:13:22.109330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.109274 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" event={"ID":"e27fc31e-f736-47f6-b5c2-f1e3e948f44e","Type":"ContainerDied","Data":"696fcc7549ed0b41be506258c68b79e1001ccf699ce10d80e347f2a538bd3c30"} Nov 03 08:13:22.124274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.123883 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:13:22.266636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.266265 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:13:22.285160 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.285131 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:13:22.293836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.293803 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.294096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.293856 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.294096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294064 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.294274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294160 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.294274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294260 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.294432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294160 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.294432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294289 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.294432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294393 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294456 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294507 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294582 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294692 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294724 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294766 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294817 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294873 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294944 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.294972 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.295064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295029 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295078 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295138 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs\") pod \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\" (UID: \"5ac2b7e5-0558-40a5-953c-1b3f57ce8918\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295171 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295198 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295240 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295290 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295312 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295342 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295413 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy\") pod \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\" (UID: \"a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295467 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs\") pod \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\" (UID: \"e27fc31e-f736-47f6-b5c2-f1e3e948f44e\") " Nov 03 08:13:22.295618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.295607 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296060 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296373 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296609 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296636 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296716 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296733 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296747 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296763 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296884 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.296958 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.297458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.297393 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.298199 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.298170 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.308694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.308484 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.314366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.310410 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:13:22.314366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.310690 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.314366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.311736 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.314839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.314797 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.317222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.317006 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:22.317222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.317130 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "kube-api-access-9f6bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:22.317222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.317130 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.317222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.317185 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:22.317222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.317193 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:13:22.317222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.317204 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.318156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.318113 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "kube-api-access-jnw9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:22.318358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.318334 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:13:22.318713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.318691 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo" pod "5ac2b7e5-0558-40a5-953c-1b3f57ce8918" (UID: "5ac2b7e5-0558-40a5-953c-1b3f57ce8918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:13:22.318810 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.318726 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6" pod "e27fc31e-f736-47f6-b5c2-f1e3e948f44e" (UID: "e27fc31e-f736-47f6-b5c2-f1e3e948f44e"). InnerVolumeSpecName "kube-api-access-fcvk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:22.319089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.319069 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:22.319156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.319117 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo" pod "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" (UID: "a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:13:22.398479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398440 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6bl\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-kube-api-access-9f6bl\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398482 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398497 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398511 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398525 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398537 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398550 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398564 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398577 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398590 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnw9s\" (UniqueName: \"kubernetes.io/projected/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-kube-api-access-jnw9s\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398604 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398618 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398631 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398663 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398678 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ac2b7e5-0558-40a5-953c-1b3f57ce8918-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398693 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398706 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398720 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvk6\" (UniqueName: \"kubernetes.io/projected/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-kube-api-access-fcvk6\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398736 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398751 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398764 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:22.398787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:22.398778 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27fc31e-f736-47f6-b5c2-f1e3e948f44e-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:23.120491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.120460 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" Nov 03 08:13:23.120726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.120463 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5" event={"ID":"5ac2b7e5-0558-40a5-953c-1b3f57ce8918","Type":"ContainerDied","Data":"d9fc53ccd5abe24b8ca1ffe5410ed3e51151bb3abb10bf4c2e1698a45ce41f0b"} Nov 03 08:13:23.120726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.120616 5951 scope.go:122] "RemoveContainer" containerID="f7806c0114ac1f2913816750d0c8d35e7ae156af88e19b5dfa67ad48e89d0c22" Nov 03 08:13:23.122563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.122534 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" event={"ID":"e27fc31e-f736-47f6-b5c2-f1e3e948f44e","Type":"ContainerDied","Data":"491d815322de5b90164763ec78ee34a7c944289bcfb3ae26333c1bb3a4375dd0"} Nov 03 08:13:23.122697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.122547 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz" Nov 03 08:13:23.122820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.122800 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5" Nov 03 08:13:23.140506 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.140483 5951 scope.go:122] "RemoveContainer" containerID="696fcc7549ed0b41be506258c68b79e1001ccf699ce10d80e347f2a538bd3c30" Nov 03 08:13:23.165592 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.165544 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5"] Nov 03 08:13:23.172404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.172373 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-m9mm5"] Nov 03 08:13:23.185950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.185912 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz"] Nov 03 08:13:23.196051 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.196018 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-crkmz"] Nov 03 08:13:23.218176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.218136 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5"] Nov 03 08:13:23.221524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.221490 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-zkps5"] Nov 03 08:13:23.264247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:23.264214 5951 scope.go:122] "RemoveContainer" containerID="aa9656116cb1d99b3502145e020939c547c95ba18b1315bd815adea2912148ba" Nov 03 08:13:24.758489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:24.758265 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" path="/var/lib/kubelet/pods/5ac2b7e5-0558-40a5-953c-1b3f57ce8918/volumes" Nov 03 08:13:24.759256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:24.759109 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" path="/var/lib/kubelet/pods/a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf/volumes" Nov 03 08:13:24.759827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:24.759808 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" path="/var/lib/kubelet/pods/e27fc31e-f736-47f6-b5c2-f1e3e948f44e/volumes" Nov 03 08:13:26.009346 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.009235 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8"] Nov 03 08:13:26.010429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.009490 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" podUID="690f04a1-abec-447c-9338-e65c4eb34a4d" containerName="maas-api" containerID="cri-o://9569c49d1d3eefa8e2c96899024898e34cffb8bf18de590bf9ac21e381997972" gracePeriod=30 Nov 03 08:13:26.156905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.156870 5951 generic.go:356] "Generic (PLEG): container finished" podID="690f04a1-abec-447c-9338-e65c4eb34a4d" containerID="9569c49d1d3eefa8e2c96899024898e34cffb8bf18de590bf9ac21e381997972" exitCode=0 Nov 03 08:13:26.157096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.156942 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" event={"ID":"690f04a1-abec-447c-9338-e65c4eb34a4d","Type":"ContainerDied","Data":"9569c49d1d3eefa8e2c96899024898e34cffb8bf18de590bf9ac21e381997972"} Nov 03 08:13:26.646098 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.645739 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:13:26.679524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.679473 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs\") pod \"690f04a1-abec-447c-9338-e65c4eb34a4d\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " Nov 03 08:13:26.679928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.679906 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\") pod \"690f04a1-abec-447c-9338-e65c4eb34a4d\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " Nov 03 08:13:26.680094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.680076 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk\" (UniqueName: \"kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk\") pod \"690f04a1-abec-447c-9338-e65c4eb34a4d\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " Nov 03 08:13:26.680535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.680504 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca\") pod \"690f04a1-abec-447c-9338-e65c4eb34a4d\" (UID: \"690f04a1-abec-447c-9338-e65c4eb34a4d\") " Nov 03 08:13:26.681918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.681894 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca" pod "690f04a1-abec-447c-9338-e65c4eb34a4d" (UID: "690f04a1-abec-447c-9338-e65c4eb34a4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:13:26.693255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.693221 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs" pod "690f04a1-abec-447c-9338-e65c4eb34a4d" (UID: "690f04a1-abec-447c-9338-e65c4eb34a4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:13:26.693614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.693589 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls" pod "690f04a1-abec-447c-9338-e65c4eb34a4d" (UID: "690f04a1-abec-447c-9338-e65c4eb34a4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:13:26.701838 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.695616 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk" pod "690f04a1-abec-447c-9338-e65c4eb34a4d" (UID: "690f04a1-abec-447c-9338-e65c4eb34a4d"). InnerVolumeSpecName "kube-api-access-sz9mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:13:26.793910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.791022 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690f04a1-abec-447c-9338-e65c4eb34a4d-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:26.793910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.791049 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:26.793910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.791065 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690f04a1-abec-447c-9338-e65c4eb34a4d-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:26.793910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:26.791081 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9mk\" (UniqueName: \"kubernetes.io/projected/690f04a1-abec-447c-9338-e65c4eb34a4d-kube-api-access-sz9mk\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:13:27.167691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:27.167660 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" Nov 03 08:13:27.167691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:27.167673 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8" event={"ID":"690f04a1-abec-447c-9338-e65c4eb34a4d","Type":"ContainerDied","Data":"b8453c9bd2b64da64efd2df909ddd880280b7be70e41f8fb47c4048930a87359"} Nov 03 08:13:27.168159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:27.167739 5951 scope.go:122] "RemoveContainer" containerID="9569c49d1d3eefa8e2c96899024898e34cffb8bf18de590bf9ac21e381997972" Nov 03 08:13:27.188319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:27.188292 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8"] Nov 03 08:13:27.192538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:27.192513 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2a44ba-86d456d56f-9p2w8"] Nov 03 08:13:28.744184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:28.744140 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690f04a1-abec-447c-9338-e65c4eb34a4d" path="/var/lib/kubelet/pods/690f04a1-abec-447c-9338-e65c4eb34a4d/volumes" Nov 03 08:13:30.077447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.077410 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:13:30.077952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.077659 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:13:30.688036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.687999 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k"] Nov 03 08:13:30.689260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689238 5951 cpu_manager.go:414] "containerMap: removing container" podUID="690f04a1-abec-447c-9338-e65c4eb34a4d" containerName="maas-api" Nov 03 08:13:30.689358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689263 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690f04a1-abec-447c-9338-e65c4eb34a4d" containerName="maas-api" Nov 03 08:13:30.689358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689292 5951 cpu_manager.go:414] "containerMap: removing container" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" Nov 03 08:13:30.689358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689300 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" Nov 03 08:13:30.689358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689314 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" Nov 03 08:13:30.689358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689321 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" Nov 03 08:13:30.689531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689409 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" Nov 03 08:13:30.689531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689417 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" Nov 03 08:13:30.689610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689596 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="690f04a1-abec-447c-9338-e65c4eb34a4d" containerName="maas-api" Nov 03 08:13:30.689680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689620 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27fc31e-f736-47f6-b5c2-f1e3e948f44e" containerName="istio-proxy" Nov 03 08:13:30.689680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689631 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac2b7e5-0558-40a5-953c-1b3f57ce8918" containerName="istio-proxy" Nov 03 08:13:30.689680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.689671 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d49e8d-1e4a-4397-b8c4-9169b3fdb4cf" containerName="istio-proxy" Nov 03 08:13:30.694629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.694602 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.727310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.727278 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb887217-serving-cert\"" Nov 03 08:13:30.765393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.765321 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.765825 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.765789 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.766321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.765860 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmr2\" (UniqueName: \"kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.766321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.765916 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.770348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.770279 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k"] Nov 03 08:13:30.867202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.867167 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.867402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.867218 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmr2\" (UniqueName: \"kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.867402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.867250 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.867402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.867286 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.868035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.867999 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.870192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.870164 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.870285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.870233 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:30.882273 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:30.882246 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmr2\" (UniqueName: \"kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2\") pod \"maas-api-e2e-mt-cb887217-7f768c896b-7c85k\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:31.016562 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:31.016462 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:31.348827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:31.348794 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k"] Nov 03 08:13:31.352100 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:13:31.352068 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7d95e1_a48b_4787_a517_cf8c0c080916.slice/crio-7241539f0c3b64dbf59d080d5f51fa01793e84ed554ab7e9c25849f4cbdd9c6e WatchSource:0}: Error finding container 7241539f0c3b64dbf59d080d5f51fa01793e84ed554ab7e9c25849f4cbdd9c6e: Status 404 returned error can't find the container with id 7241539f0c3b64dbf59d080d5f51fa01793e84ed554ab7e9c25849f4cbdd9c6e Nov 03 08:13:31.999692 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:13:31.999655 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:13:32.113099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:32.113020 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:13:32.219638 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:32.219599 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" event={"ID":"ae7d95e1-a48b-4787-a517-cf8c0c080916","Type":"ContainerStarted","Data":"82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50"} Nov 03 08:13:32.219870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:32.219710 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" event={"ID":"ae7d95e1-a48b-4787-a517-cf8c0c080916","Type":"ContainerStarted","Data":"7241539f0c3b64dbf59d080d5f51fa01793e84ed554ab7e9c25849f4cbdd9c6e"} Nov 03 08:13:32.219870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:32.219753 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:32.242134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:32.242086 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" podStartSLOduration=1.733945061 podStartE2EDuration="2.242071486s" totalImagesPullingTime="508.126425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:32.237685381 +0000 UTC m=+1151.619136288" watchObservedRunningTime="2026-11-03 08:13:32.242071486 +0000 UTC m=+1151.623522406" Nov 03 08:13:38.240292 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.240262 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:13:38.493584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.493503 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz"] Nov 03 08:13:38.507012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.506982 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.523026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.522996 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-94e3cb-serving-cert\"" Nov 03 08:13:38.531684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.531259 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz"] Nov 03 08:13:38.664229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.664185 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.664552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.664511 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.664798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.664783 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.664936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.664907 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqt7\" (UniqueName: \"kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.766464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.766358 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.766661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.766570 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.766752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.766693 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqt7\" (UniqueName: \"kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.766910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.766893 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.771823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.771789 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.773544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.773516 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.774280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.774258 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.777714 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.777631 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqt7\" (UniqueName: \"kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7\") pod \"maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:38.836858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:38.836823 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:39.249000 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:13:39.248940 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06b5868_573b_4972_b1fd_9c6f681bf3aa.slice/crio-a4add3d8686a128db1ccce92e82f3640b07993cbed190c2b8e3f689a692408d3 WatchSource:0}: Error finding container a4add3d8686a128db1ccce92e82f3640b07993cbed190c2b8e3f689a692408d3: Status 404 returned error can't find the container with id a4add3d8686a128db1ccce92e82f3640b07993cbed190c2b8e3f689a692408d3 Nov 03 08:13:39.268628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:39.260362 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:13:39.276237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:39.275051 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz"] Nov 03 08:13:39.369670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:39.366534 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" event={"ID":"d06b5868-573b-4972-b1fd-9c6f681bf3aa","Type":"ContainerStarted","Data":"a4add3d8686a128db1ccce92e82f3640b07993cbed190c2b8e3f689a692408d3"} Nov 03 08:13:40.380127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:40.380075 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" event={"ID":"d06b5868-573b-4972-b1fd-9c6f681bf3aa","Type":"ContainerStarted","Data":"e91dba1a1172d4222710a63eb28405599780a26264e5e9954adb6c2a27a6f245"} Nov 03 08:13:40.380607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:40.380155 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:13:40.399825 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:40.399760 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" podStartSLOduration=1.730762115 podStartE2EDuration="2.399742509s" totalImagesPullingTime="668.980394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:40.396811265 +0000 UTC m=+1159.778262183" watchObservedRunningTime="2026-11-03 08:13:40.399742509 +0000 UTC m=+1159.781193427" Nov 03 08:13:46.400031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:13:46.399991 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:14:01.959747 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:01.959707 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:14:05.823636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:05.823554 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh"] Nov 03 08:14:05.824454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:05.824422 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" podUID="96fce170-8828-446a-ba55-b3d2b97864da" containerName="maas-api" containerID="cri-o://f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83" gracePeriod=30 Nov 03 08:14:05.866745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:05.866494 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb"] Nov 03 08:14:05.876442 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:05.876411 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:05.877621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:05.877592 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb"] Nov 03 08:14:05.887340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:05.887265 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dfb883-openshift-default-dockercfg-mtfb5\"" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024239 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024293 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024335 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024372 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024389 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024458 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hmm\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024780 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024868 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.026705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.024969 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132183 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132248 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132304 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132359 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132385 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132476 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hmm\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132623 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132738 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.139726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.132842 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.171670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.142304 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.172709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.172679 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.271770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.271725 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.273271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.272373 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.273462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.272976 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.392303 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.392110 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.392780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.392732 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.402441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.402379 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hmm\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.426423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.426266 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.446219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.446186 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:14:06.521469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.521315 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\") pod \"96fce170-8828-446a-ba55-b3d2b97864da\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " Nov 03 08:14:06.521905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.521883 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca\") pod \"96fce170-8828-446a-ba55-b3d2b97864da\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " Nov 03 08:14:06.522245 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.522211 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7\" (UniqueName: \"kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7\") pod \"96fce170-8828-446a-ba55-b3d2b97864da\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " Nov 03 08:14:06.522442 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.522427 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs\") pod \"96fce170-8828-446a-ba55-b3d2b97864da\" (UID: \"96fce170-8828-446a-ba55-b3d2b97864da\") " Nov 03 08:14:06.523005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.522281 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca" pod "96fce170-8828-446a-ba55-b3d2b97864da" (UID: "96fce170-8828-446a-ba55-b3d2b97864da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:06.525020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.524989 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96fce170-8828-446a-ba55-b3d2b97864da-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:06.534758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.530147 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls" pod "96fce170-8828-446a-ba55-b3d2b97864da" (UID: "96fce170-8828-446a-ba55-b3d2b97864da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:14:06.534758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.530463 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:06.534758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.531189 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs" pod "96fce170-8828-446a-ba55-b3d2b97864da" (UID: "96fce170-8828-446a-ba55-b3d2b97864da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:14:06.534758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.531718 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7" pod "96fce170-8828-446a-ba55-b3d2b97864da" (UID: "96fce170-8828-446a-ba55-b3d2b97864da"). InnerVolumeSpecName "kube-api-access-dgcs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:06.629436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.628475 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcs7\" (UniqueName: \"kubernetes.io/projected/96fce170-8828-446a-ba55-b3d2b97864da-kube-api-access-dgcs7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:06.629436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.628510 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:06.629436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.628529 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96fce170-8828-446a-ba55-b3d2b97864da-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:06.771327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.769669 5951 generic.go:356] "Generic (PLEG): container finished" podID="96fce170-8828-446a-ba55-b3d2b97864da" containerID="f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83" exitCode=0 Nov 03 08:14:06.771327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.769817 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" event={"ID":"96fce170-8828-446a-ba55-b3d2b97864da","Type":"ContainerDied","Data":"f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83"} Nov 03 08:14:06.771327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.769929 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" event={"ID":"96fce170-8828-446a-ba55-b3d2b97864da","Type":"ContainerDied","Data":"76910e0d44ea1403116feeeb413f388316e0a2768f69fe8372c720c8618b6bb2"} Nov 03 08:14:06.771327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.769988 5951 scope.go:122] "RemoveContainer" containerID="f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83" Nov 03 08:14:06.771327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.770159 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh" Nov 03 08:14:06.855635 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.853249 5951 scope.go:122] "RemoveContainer" containerID="f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83" Nov 03 08:14:06.859256 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:06.859216 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83\": container with ID starting with f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83 not found: ID does not exist" containerID="f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83" Nov 03 08:14:06.859390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.859267 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83"} err="failed to get container status \"f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83\": rpc error: code = NotFound desc = could not find container \"f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83\": container with ID starting with f9548d10504137798e7a11ed6dcaebe5f535a71d97c47b5cfc49f378527a7f83 not found: ID does not exist" Nov 03 08:14:06.874759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.874072 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh"] Nov 03 08:14:06.879441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.879398 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dfb883-5d8c8b9d95-792vh"] Nov 03 08:14:06.890183 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.890149 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb"] Nov 03 08:14:06.900985 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:06.900913 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc5f08a_832b_40c3_9309_fb30e26b7ae8.slice/crio-fdbd4276a25f347d16473b9618fda36dcadf8a5a17648f814359e3180dcbfb72 WatchSource:0}: Error finding container fdbd4276a25f347d16473b9618fda36dcadf8a5a17648f814359e3180dcbfb72: Status 404 returned error can't find the container with id fdbd4276a25f347d16473b9618fda36dcadf8a5a17648f814359e3180dcbfb72 Nov 03 08:14:06.904828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.904786 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:06.904938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.904876 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:06.904938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:06.904921 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:07.784903 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:07.784867 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" event={"ID":"ccc5f08a-832b-40c3-9309-fb30e26b7ae8","Type":"ContainerStarted","Data":"52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001"} Nov 03 08:14:07.785085 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:07.784928 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" event={"ID":"ccc5f08a-832b-40c3-9309-fb30e26b7ae8","Type":"ContainerStarted","Data":"fdbd4276a25f347d16473b9618fda36dcadf8a5a17648f814359e3180dcbfb72"} Nov 03 08:14:07.806825 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:07.806737 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podStartSLOduration=2.806719131 podStartE2EDuration="2.806719131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:07.80443843 +0000 UTC m=+1187.185889365" watchObservedRunningTime="2026-11-03 08:14:07.806719131 +0000 UTC m=+1187.188170070" Nov 03 08:14:08.196031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:08.195945 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9"] Nov 03 08:14:08.208829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:08.208794 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62"] Nov 03 08:14:08.544734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:08.538834 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:08.544734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:08.544612 5951 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 08:14:08.545109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:08.545078 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 08:14:08.756741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:08.749707 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fce170-8828-446a-ba55-b3d2b97864da" path="/var/lib/kubelet/pods/96fce170-8828-446a-ba55-b3d2b97864da/volumes" Nov 03 08:14:09.012053 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.011966 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d"] Nov 03 08:14:09.012870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.012852 5951 cpu_manager.go:414] "containerMap: removing container" podUID="96fce170-8828-446a-ba55-b3d2b97864da" containerName="maas-api" Nov 03 08:14:09.012870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.012869 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fce170-8828-446a-ba55-b3d2b97864da" containerName="maas-api" Nov 03 08:14:09.013041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.013030 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fce170-8828-446a-ba55-b3d2b97864da" containerName="maas-api" Nov 03 08:14:09.017358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.017324 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.032599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.032117 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-228695-dockercfg-ctdjj\"" Nov 03 08:14:09.032599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.032204 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-228695\"" Nov 03 08:14:09.047713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.047628 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.047875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.047738 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jrw\" (UniqueName: \"kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw\") pod \"payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.058833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.058807 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d"] Nov 03 08:14:09.075065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.075030 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl"] Nov 03 08:14:09.096737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.095861 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.171046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.144703 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl"] Nov 03 08:14:09.205671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.177220 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgbs\" (UniqueName: \"kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs\") pod \"payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.205671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.177300 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume\") pod \"payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.205671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.177338 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.205671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.177414 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jrw\" (UniqueName: \"kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw\") pod \"payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.210671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.206544 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.253668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.244634 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jrw\" (UniqueName: \"kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw\") pod \"payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.365931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.279027 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgbs\" (UniqueName: \"kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs\") pod \"payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.365931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.279117 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume\") pod \"payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.365931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.280537 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume\") pod \"payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.365931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.345869 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:09.503125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.503049 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgbs\" (UniqueName: \"kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs\") pod \"payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.532214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.532182 5951 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 08:14:09.532475 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.532449 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 08:14:09.634298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.616961 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:09.881160 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:09.881125 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n"] Nov 03 08:14:10.106997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.102099 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d"] Nov 03 08:14:10.183388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.183356 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl"] Nov 03 08:14:10.541709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.541315 5951 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 08:14:10.541709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.541458 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 08:14:10.866247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.866208 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" event={"ID":"d9a064df-d861-461f-8412-b6c7d05d4aa8","Type":"ContainerStarted","Data":"f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f"} Nov 03 08:14:10.866419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.866287 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" event={"ID":"d9a064df-d861-461f-8412-b6c7d05d4aa8","Type":"ContainerStarted","Data":"075b6e296126f8492d0cb121dadb1d24a8ee0832cc99fb46ead9bacce2165646"} Nov 03 08:14:10.866419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.866315 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:10.867923 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.867895 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" event={"ID":"8ec928ec-d13e-4510-a8f2-21776ab47979","Type":"ContainerStarted","Data":"47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53"} Nov 03 08:14:10.868040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.867938 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" event={"ID":"8ec928ec-d13e-4510-a8f2-21776ab47979","Type":"ContainerStarted","Data":"a7f3fb40b4b2c44e4c3120e9e4b98367219418dd8b7a471f5eb840ac92928dcd"} Nov 03 08:14:10.868040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.867958 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:10.885387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.885343 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" podStartSLOduration=2.8853283210000003 podStartE2EDuration="2.885328321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:10.881466019 +0000 UTC m=+1190.262916937" watchObservedRunningTime="2026-11-03 08:14:10.885328321 +0000 UTC m=+1190.266779238" Nov 03 08:14:10.922721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:10.922070 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" podStartSLOduration=1.922051832 podStartE2EDuration="1.922051832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:10.910970434 +0000 UTC m=+1190.292421364" watchObservedRunningTime="2026-11-03 08:14:10.922051832 +0000 UTC m=+1190.303502751" Nov 03 08:14:11.531720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:11.531682 5951 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 08:14:11.531922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:11.531754 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 08:14:12.531587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:12.531537 5951 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 08:14:12.532028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:12.531601 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 08:14:13.220552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.215715 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" podUID="60394526-fb1e-4d6f-8321-f89db0abcc29" containerName="payload-pre-processing" containerID="cri-o://990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e" gracePeriod=25 Nov 03 08:14:13.225857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.225807 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" podUID="8aab0f91-f6ac-4938-a5bb-273bddbe8346" containerName="payload-processing" containerID="cri-o://da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca" gracePeriod=25 Nov 03 08:14:13.532611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.532145 5951 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 08:14:13.532611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.532209 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 08:14:13.655458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.655412 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:14:13.753716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.738769 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:14:13.783813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.781585 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume\") pod \"60394526-fb1e-4d6f-8321-f89db0abcc29\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " Nov 03 08:14:13.783813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.782159 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8\" (UniqueName: \"kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8\") pod \"60394526-fb1e-4d6f-8321-f89db0abcc29\" (UID: \"60394526-fb1e-4d6f-8321-f89db0abcc29\") " Nov 03 08:14:13.783813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.783533 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume" pod "60394526-fb1e-4d6f-8321-f89db0abcc29" (UID: "60394526-fb1e-4d6f-8321-f89db0abcc29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:13.784102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.783983 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60394526-fb1e-4d6f-8321-f89db0abcc29-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:13.790181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.788382 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8" pod "60394526-fb1e-4d6f-8321-f89db0abcc29" (UID: "60394526-fb1e-4d6f-8321-f89db0abcc29"). InnerVolumeSpecName "kube-api-access-jf9j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:13.885393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.885362 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt\" (UniqueName: \"kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt\") pod \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " Nov 03 08:14:13.885590 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.885441 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume\") pod \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\" (UID: \"8aab0f91-f6ac-4938-a5bb-273bddbe8346\") " Nov 03 08:14:13.886496 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.886247 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9j8\" (UniqueName: \"kubernetes.io/projected/60394526-fb1e-4d6f-8321-f89db0abcc29-kube-api-access-jf9j8\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:13.888106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.886766 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume" pod "8aab0f91-f6ac-4938-a5bb-273bddbe8346" (UID: "8aab0f91-f6ac-4938-a5bb-273bddbe8346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:13.888450 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.888428 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt" pod "8aab0f91-f6ac-4938-a5bb-273bddbe8346" (UID: "8aab0f91-f6ac-4938-a5bb-273bddbe8346"). InnerVolumeSpecName "kube-api-access-lkrwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:13.965635 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.965592 5951 generic.go:356] "Generic (PLEG): container finished" podID="8aab0f91-f6ac-4938-a5bb-273bddbe8346" containerID="da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca" exitCode=0 Nov 03 08:14:13.965874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.965827 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" event={"ID":"8aab0f91-f6ac-4938-a5bb-273bddbe8346","Type":"ContainerDied","Data":"da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca"} Nov 03 08:14:13.965938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.965921 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" event={"ID":"8aab0f91-f6ac-4938-a5bb-273bddbe8346","Type":"ContainerDied","Data":"14a8f3a404d08f2be8b41ff49ffd60eccfbeec02efab21f3897ca61c1cf54c2a"} Nov 03 08:14:13.965977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.965960 5951 scope.go:122] "RemoveContainer" containerID="da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca" Nov 03 08:14:13.966164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.966150 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9" Nov 03 08:14:13.971443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.971381 5951 generic.go:356] "Generic (PLEG): container finished" podID="60394526-fb1e-4d6f-8321-f89db0abcc29" containerID="990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e" exitCode=0 Nov 03 08:14:13.971637 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.971615 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" Nov 03 08:14:13.971775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.971749 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" event={"ID":"60394526-fb1e-4d6f-8321-f89db0abcc29","Type":"ContainerDied","Data":"990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e"} Nov 03 08:14:13.971841 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.971811 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62" event={"ID":"60394526-fb1e-4d6f-8321-f89db0abcc29","Type":"ContainerDied","Data":"c284521d95955c86ccb68477784b47fbf0365ef7d3503a7e8d1c78bd16e9bb8c"} Nov 03 08:14:13.984836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.984817 5951 scope.go:122] "RemoveContainer" containerID="da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca" Nov 03 08:14:13.985129 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:13.985104 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca\": container with ID starting with da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca not found: ID does not exist" containerID="da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca" Nov 03 08:14:13.985185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.985139 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca"} err="failed to get container status \"da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca\": rpc error: code = NotFound desc = could not find container \"da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca\": container with ID starting with da6140e49c1c6f37ebbbbd3fd5bb5382386c5fb117250020dfa112a546589dca not found: ID does not exist" Nov 03 08:14:13.985185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.985156 5951 scope.go:122] "RemoveContainer" containerID="990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e" Nov 03 08:14:13.988784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.988759 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkrwt\" (UniqueName: \"kubernetes.io/projected/8aab0f91-f6ac-4938-a5bb-273bddbe8346-kube-api-access-lkrwt\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:13.988784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:13.988785 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aab0f91-f6ac-4938-a5bb-273bddbe8346-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:14.037965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.037893 5951 scope.go:122] "RemoveContainer" containerID="990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e" Nov 03 08:14:14.039721 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:14.039395 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e\": container with ID starting with 990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e not found: ID does not exist" containerID="990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e" Nov 03 08:14:14.039721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.039438 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e"} err="failed to get container status \"990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e\": rpc error: code = NotFound desc = could not find container \"990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e\": container with ID starting with 990c796192efb0369d9d56ff9a77f76062761f4144d6697960ce03667b11e46e not found: ID does not exist" Nov 03 08:14:14.042962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.042935 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9"] Nov 03 08:14:14.050375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.050336 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb887217-5b45949c5f-f7ll9"] Nov 03 08:14:14.054863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.054841 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62"] Nov 03 08:14:14.061766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.061358 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb887217-689fcf7555-d6q62"] Nov 03 08:14:14.186071 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.185230 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb"] Nov 03 08:14:14.186071 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.185469 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" containerID="cri-o://52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001" gracePeriod=30 Nov 03 08:14:14.757672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.757587 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60394526-fb1e-4d6f-8321-f89db0abcc29" path="/var/lib/kubelet/pods/60394526-fb1e-4d6f-8321-f89db0abcc29/volumes" Nov 03 08:14:14.759965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.758424 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aab0f91-f6ac-4938-a5bb-273bddbe8346" path="/var/lib/kubelet/pods/8aab0f91-f6ac-4938-a5bb-273bddbe8346/volumes" Nov 03 08:14:14.882483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:14.882428 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" podUID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" containerName="payload-pre-processing" containerID="cri-o://35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756" gracePeriod=25 Nov 03 08:14:15.150714 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:15.149278 5951 logging.go:55] [core] [Channel #2222 SubChannel #2223]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.112:50052", ServerName: "10.134.0.112:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.112:50052: connect: connection refused" Nov 03 08:14:15.378838 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.378807 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:14:15.390635 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.390603 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d"] Nov 03 08:14:15.391927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.391897 5951 cpu_manager.go:414] "containerMap: removing container" podUID="8aab0f91-f6ac-4938-a5bb-273bddbe8346" containerName="payload-processing" Nov 03 08:14:15.391927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.391926 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aab0f91-f6ac-4938-a5bb-273bddbe8346" containerName="payload-processing" Nov 03 08:14:15.392110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.391978 5951 cpu_manager.go:414] "containerMap: removing container" podUID="60394526-fb1e-4d6f-8321-f89db0abcc29" containerName="payload-pre-processing" Nov 03 08:14:15.392110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.391986 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60394526-fb1e-4d6f-8321-f89db0abcc29" containerName="payload-pre-processing" Nov 03 08:14:15.392110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.392029 5951 cpu_manager.go:414] "containerMap: removing container" podUID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" containerName="payload-pre-processing" Nov 03 08:14:15.392110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.392036 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" containerName="payload-pre-processing" Nov 03 08:14:15.392305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.392249 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" containerName="payload-pre-processing" Nov 03 08:14:15.392305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.392269 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aab0f91-f6ac-4938-a5bb-273bddbe8346" containerName="payload-processing" Nov 03 08:14:15.392305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.392288 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="60394526-fb1e-4d6f-8321-f89db0abcc29" containerName="payload-pre-processing" Nov 03 08:14:15.396887 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.396861 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.438228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.436236 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d"] Nov 03 08:14:15.439222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.439161 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ba69a0-openshift-default-dockercfg-pdjrt\"" Nov 03 08:14:15.456292 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.456258 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.457118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.457086 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.457328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.457307 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.457535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.457517 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.457794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.457774 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.457932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.457915 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.458055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.458040 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.458185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.458168 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27x8h\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.458370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.458352 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.565937 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.565895 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c\" (UniqueName: \"kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c\") pod \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " Nov 03 08:14:15.566141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.566014 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume\") pod \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\" (UID: \"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e\") " Nov 03 08:14:15.566509 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.566377 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume" pod "3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" (UID: "3cccd504-3dc2-4c0b-af6b-8e08cb5a180e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:15.566509 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.566431 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.567865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.567790 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.568869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.568164 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.569187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.569160 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.573232 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.572925 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c" pod "3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" (UID: "3cccd504-3dc2-4c0b-af6b-8e08cb5a180e"). InnerVolumeSpecName "kube-api-access-xqr6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:15.590798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.579308 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.599672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.591266 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.600087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.600049 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.615812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.615780 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.616075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.616055 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27x8h\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.616215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.616190 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.616350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.616335 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.616581 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.616564 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.617153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.617130 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.617308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.617291 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqr6c\" (UniqueName: \"kubernetes.io/projected/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-kube-api-access-xqr6c\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:15.641669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.640553 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:15.641669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.618528 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.641669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.640503 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.641669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.617985 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.675667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.660228 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.675667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.660903 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27x8h\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:15.740667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:15.739209 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:16.000371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.000336 5951 generic.go:356] "Generic (PLEG): container finished" podID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" containerID="35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756" exitCode=0 Nov 03 08:14:16.001971 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.000885 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" Nov 03 08:14:16.004455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.004424 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" event={"ID":"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e","Type":"ContainerDied","Data":"35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756"} Nov 03 08:14:16.004565 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.004488 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" event={"ID":"3cccd504-3dc2-4c0b-af6b-8e08cb5a180e","Type":"ContainerDied","Data":"4f8992562abd882feb141b1c036374101e404dadab62a0226498dff047e68b5f"} Nov 03 08:14:16.004565 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.004521 5951 scope.go:122] "RemoveContainer" containerID="35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756" Nov 03 08:14:16.051754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.051379 5951 scope.go:122] "RemoveContainer" containerID="35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756" Nov 03 08:14:16.052222 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:16.052032 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756\": container with ID starting with 35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756 not found: ID does not exist" containerID="35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756" Nov 03 08:14:16.052222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.052070 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756"} err="failed to get container status \"35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756\": rpc error: code = NotFound desc = could not find container \"35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756\": container with ID starting with 35db263fa354e7ec8ddb46fe3f40c9e4befcbfcbf282a73e7966ed27755fe756 not found: ID does not exist" Nov 03 08:14:16.059873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.058365 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n"] Nov 03 08:14:16.065298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.065237 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n"] Nov 03 08:14:16.131827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.131783 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e255cbee-846944cdbc-q9d8n" podUID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.112:50052\" within 1s: context deadline exceeded" Nov 03 08:14:16.281235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.281207 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d"] Nov 03 08:14:16.281791 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.281763 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:16.281865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.281822 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:16.281865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.281850 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:16.410462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.410423 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99"] Nov 03 08:14:16.410717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.410688 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" containerID="cri-o://9ecd7b0ddadff368de7d0854e0b46d7c6b5a846757833ffa3da46bb4667cbc25" gracePeriod=30 Nov 03 08:14:16.418081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.418054 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp"] Nov 03 08:14:16.418305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.418277 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" containerID="cri-o://898a0b2640550e79c5f947d656a900bb8259c21c21d33389a2f258c6864390d4" gracePeriod=30 Nov 03 08:14:16.758707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.755599 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cccd504-3dc2-4c0b-af6b-8e08cb5a180e" path="/var/lib/kubelet/pods/3cccd504-3dc2-4c0b-af6b-8e08cb5a180e/volumes" Nov 03 08:14:16.840392 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.840351 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj"] Nov 03 08:14:16.840896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.840861 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" containerID="cri-o://5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba" gracePeriod=30 Nov 03 08:14:16.851519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.851485 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v"] Nov 03 08:14:16.851783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:16.851747 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" containerID="cri-o://61a6301ddd5247d62149a43aa3964451af25105f0cfa415b7210c11b8404739c" gracePeriod=30 Nov 03 08:14:17.016540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:17.016406 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" event={"ID":"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5","Type":"ContainerStarted","Data":"5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62"} Nov 03 08:14:17.016540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:17.016476 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" event={"ID":"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5","Type":"ContainerStarted","Data":"9fb525d8ac99c2b46b75254fcbbb1ddb897acb1bb72dad52749dbb426d23c270"} Nov 03 08:14:17.040060 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:17.039999 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podStartSLOduration=2.03997989 podStartE2EDuration="2.03997989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:17.037523303 +0000 UTC m=+1196.418974219" watchObservedRunningTime="2026-11-03 08:14:17.03997989 +0000 UTC m=+1196.421430807" Nov 03 08:14:17.740180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:17.740147 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:17.746703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:17.746632 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:18.026833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:18.026800 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:18.027982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:18.027964 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:14:20.958695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:20.958344 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:20.976674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:20.976597 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8"] Nov 03 08:14:20.977973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:20.977944 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" Nov 03 08:14:20.977973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:20.977973 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" Nov 03 08:14:20.978265 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:20.978247 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerName="istio-proxy" Nov 03 08:14:20.983758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:20.983734 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.012081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.012050 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ba69a0\"" Nov 03 08:14:21.012241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.012184 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ba69a0-dockercfg-wknxq\"" Nov 03 08:14:21.027668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.025770 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8"] Nov 03 08:14:21.065926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.065892 5951 generic.go:356] "Generic (PLEG): container finished" podID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" containerID="52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001" exitCode=0 Nov 03 08:14:21.066153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.065968 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" event={"ID":"ccc5f08a-832b-40c3-9309-fb30e26b7ae8","Type":"ContainerDied","Data":"52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001"} Nov 03 08:14:21.066153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.066040 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" event={"ID":"ccc5f08a-832b-40c3-9309-fb30e26b7ae8","Type":"ContainerDied","Data":"fdbd4276a25f347d16473b9618fda36dcadf8a5a17648f814359e3180dcbfb72"} Nov 03 08:14:21.066153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.066071 5951 scope.go:122] "RemoveContainer" containerID="52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001" Nov 03 08:14:21.066298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.066217 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb" Nov 03 08:14:21.082116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.082096 5951 scope.go:122] "RemoveContainer" containerID="52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001" Nov 03 08:14:21.082824 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:21.082757 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001\": container with ID starting with 52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001 not found: ID does not exist" containerID="52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001" Nov 03 08:14:21.082824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.082788 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001"} err="failed to get container status \"52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001\": rpc error: code = NotFound desc = could not find container \"52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001\": container with ID starting with 52a5f0868f172e88ed1338913c725209d735599750b3a0a5c077f863233df001 not found: ID does not exist" Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.151744 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.151856 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.151956 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.152003 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.152094 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.152153 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.152211 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.152385 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.152482 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm\") pod \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\" (UID: \"ccc5f08a-832b-40c3-9309-fb30e26b7ae8\") " Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.153005 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.153291 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5mf8\" (UniqueName: \"kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8\") pod \"payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.169720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.153732 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:21.206863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.204744 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:21.210366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.210277 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:21.221794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.221756 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:21.221948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.221843 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:21.222228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.222203 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl"] Nov 03 08:14:21.233784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.233760 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.241612 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.241577 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl"] Nov 03 08:14:21.350757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.325048 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:21.350757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.325133 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:21.350757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.325187 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "kube-api-access-48hmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:21.362437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.359967 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.362437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.360295 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5mf8\" (UniqueName: \"kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8\") pod \"payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.377183 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.384970 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386218 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386399 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386413 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386426 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386438 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386451 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.390491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.386464 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hmm\" (UniqueName: \"kubernetes.io/projected/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-kube-api-access-48hmm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.419683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.418911 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo" pod "ccc5f08a-832b-40c3-9309-fb30e26b7ae8" (UID: "ccc5f08a-832b-40c3-9309-fb30e26b7ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:21.463914 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.459805 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5mf8\" (UniqueName: \"kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8\") pod \"payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.504907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.492184 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtq6l\" (UniqueName: \"kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l\") pod \"payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.504907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.492575 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.504907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.492733 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc5f08a-832b-40c3-9309-fb30e26b7ae8-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:21.594217 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.593979 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtq6l\" (UniqueName: \"kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l\") pod \"payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.596870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.596835 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.598094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.597925 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.604441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.604353 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:14:21.611656 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.611352 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:14:21.614948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.614898 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ba69a0-dockercfg-wknxq\"" Nov 03 08:14:21.616076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.616051 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtq6l\" (UniqueName: \"kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l\") pod \"payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:21.627635 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.627315 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:21.916360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.873928 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7"] Nov 03 08:14:21.916360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.874192 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" podUID="0c6f0149-90e4-4644-8f93-1a5f38caff96" containerName="maas-api" containerID="cri-o://0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f" gracePeriod=30 Nov 03 08:14:21.933897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.927913 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb"] Nov 03 08:14:21.933897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.928062 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-7fdbb"] Nov 03 08:14:21.951809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.935845 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:14:21.951809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:21.944368 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:22.100739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.099585 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:14:22.201879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.201839 5951 generic.go:356] "Generic (PLEG): container finished" podID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerID="61a6301ddd5247d62149a43aa3964451af25105f0cfa415b7210c11b8404739c" exitCode=0 Nov 03 08:14:22.202304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.202280 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" event={"ID":"3c21cf61-4d1e-41a0-adef-173cce70680a","Type":"ContainerDied","Data":"61a6301ddd5247d62149a43aa3964451af25105f0cfa415b7210c11b8404739c"} Nov 03 08:14:22.208762 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.208729 5951 generic.go:356] "Generic (PLEG): container finished" podID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerID="9ecd7b0ddadff368de7d0854e0b46d7c6b5a846757833ffa3da46bb4667cbc25" exitCode=0 Nov 03 08:14:22.209120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.209095 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" event={"ID":"aa445227-7a5e-4b11-8fd6-55fdef0cc618","Type":"ContainerDied","Data":"9ecd7b0ddadff368de7d0854e0b46d7c6b5a846757833ffa3da46bb4667cbc25"} Nov 03 08:14:22.214951 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.214922 5951 generic.go:356] "Generic (PLEG): container finished" podID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerID="5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba" exitCode=0 Nov 03 08:14:22.215285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.215262 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" event={"ID":"d60e3d10-bc4e-42c9-93af-51c843e47fd9","Type":"ContainerDied","Data":"5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba"} Nov 03 08:14:22.218770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.218674 5951 generic.go:356] "Generic (PLEG): container finished" podID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerID="898a0b2640550e79c5f947d656a900bb8259c21c21d33389a2f258c6864390d4" exitCode=0 Nov 03 08:14:22.218989 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.218896 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" event={"ID":"2a52f05e-2638-4c97-ae7e-438d5a93382d","Type":"ContainerDied","Data":"898a0b2640550e79c5f947d656a900bb8259c21c21d33389a2f258c6864390d4"} Nov 03 08:14:22.894453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.893712 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:14:22.905700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.905576 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:14:22.927390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.916789 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc5f08a-832b-40c3-9309-fb30e26b7ae8" path="/var/lib/kubelet/pods/ccc5f08a-832b-40c3-9309-fb30e26b7ae8/volumes" Nov 03 08:14:22.961242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.961197 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:14:22.972494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972463 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.972710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972607 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.972710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972682 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.972849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972815 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.972908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972864 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.972908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972896 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.972941 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.973063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973024 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973117 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973074 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.973172 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973135 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.973172 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973164 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.973270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973196 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.973270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973243 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973382 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973263 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.973382 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973308 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973380 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo\") pod \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\" (UID: \"aa445227-7a5e-4b11-8fd6-55fdef0cc618\") " Nov 03 08:14:22.973488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973438 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973477 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973634 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973507 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.973634 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973506 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.973634 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973589 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data\") pod \"2a52f05e-2638-4c97-ae7e-438d5a93382d\" (UID: \"2a52f05e-2638-4c97-ae7e-438d5a93382d\") " Nov 03 08:14:22.973818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973728 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:22.973865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973834 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:22.973915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.973864 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.974338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974315 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.974449 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974366 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.974659 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974620 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974778 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974760 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a52f05e-2638-4c97-ae7e-438d5a93382d-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974783 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974798 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974813 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974835 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974850 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.974911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974863 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:22.975405 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.974913 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.975405 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.975011 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.976834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.976806 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:22.979275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.979232 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:22.979545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.979466 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.979689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.979546 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "kube-api-access-5lkdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:22.979689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.979577 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:22.979803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.979769 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy" pod "2a52f05e-2638-4c97-ae7e-438d5a93382d" (UID: "2a52f05e-2638-4c97-ae7e-438d5a93382d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:22.985301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.984361 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:22.986813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:22.986785 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr" pod "aa445227-7a5e-4b11-8fd6-55fdef0cc618" (UID: "aa445227-7a5e-4b11-8fd6-55fdef0cc618"). InnerVolumeSpecName "kube-api-access-qngfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:23.050185 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:23.050155 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f93310_b389_418c_bde6_4867568cce3b.slice/crio-d253fe6bcf1f4ef8eb56f9daaaef50427504911f1977961a13f3c147f1ac89aa WatchSource:0}: Error finding container d253fe6bcf1f4ef8eb56f9daaaef50427504911f1977961a13f3c147f1ac89aa: Status 404 returned error can't find the container with id d253fe6bcf1f4ef8eb56f9daaaef50427504911f1977961a13f3c147f1ac89aa Nov 03 08:14:23.051275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.051243 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8"] Nov 03 08:14:23.076054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.076019 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.076303 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.076281 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.076488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.076471 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.076615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.076601 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.076986 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.076793 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.077114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.077087 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.077258 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.077242 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.077765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.077743 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.077853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.077805 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk\") pod \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\" (UID: \"d60e3d10-bc4e-42c9-93af-51c843e47fd9\") " Nov 03 08:14:23.078272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.076509 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.078370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.078241 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.079018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.078997 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079126 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079215 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079304 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079395 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079385 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079472 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079459 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngfr\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-kube-api-access-qngfr\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079556 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079544 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkdx\" (UniqueName: \"kubernetes.io/projected/2a52f05e-2638-4c97-ae7e-438d5a93382d-kube-api-access-5lkdx\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079660 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079775 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079884 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079968 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.079987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079988 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa445227-7a5e-4b11-8fd6-55fdef0cc618-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.080209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079057 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:23.080209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079325 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.080209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.079840 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.080468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.080434 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.081687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.081669 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:14:23.104273 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.103815 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "kube-api-access-vgsvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:23.123133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.123060 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:23.125118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.125058 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo" pod "d60e3d10-bc4e-42c9-93af-51c843e47fd9" (UID: "d60e3d10-bc4e-42c9-93af-51c843e47fd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:23.181327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.181293 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.181471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.181341 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.181471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.181449 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:14:23.181598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.181468 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.181706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.181683 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:23.182113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.182007 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.183847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.183823 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.184006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.183889 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.184006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.183943 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.184006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.183972 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.184216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.184159 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.184288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.184238 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.184364 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.184289 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo\") pod \"3c21cf61-4d1e-41a0-adef-173cce70680a\" (UID: \"3c21cf61-4d1e-41a0-adef-173cce70680a\") " Nov 03 08:14:23.184578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.184292 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.184578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.184477 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.185055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185031 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "kube-api-access-6c85x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:23.185154 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185085 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185154 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185123 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185154 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185141 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185156 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185170 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185184 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185201 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185215 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e3d10-bc4e-42c9-93af-51c843e47fd9-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185228 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsvk\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-kube-api-access-vgsvk\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185244 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185258 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c21cf61-4d1e-41a0-adef-173cce70680a-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185271 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e3d10-bc4e-42c9-93af-51c843e47fd9-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.185845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.185623 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:23.187115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.187088 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:23.187846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.187824 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy" pod "3c21cf61-4d1e-41a0-adef-173cce70680a" (UID: "3c21cf61-4d1e-41a0-adef-173cce70680a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:23.233947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.233292 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" Nov 03 08:14:23.233947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.233344 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp" event={"ID":"2a52f05e-2638-4c97-ae7e-438d5a93382d","Type":"ContainerDied","Data":"29d602145b5ae612f8eed9cd6fee813064e229825f4d014eb3dec7d10a236ec3"} Nov 03 08:14:23.233947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.233400 5951 scope.go:122] "RemoveContainer" containerID="898a0b2640550e79c5f947d656a900bb8259c21c21d33389a2f258c6864390d4" Nov 03 08:14:23.250683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.250456 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" event={"ID":"3c21cf61-4d1e-41a0-adef-173cce70680a","Type":"ContainerDied","Data":"abf799cc6cfef5eb85afda017f0cb220ee89a7ef4eebb9a8ed9c2f6f676b1704"} Nov 03 08:14:23.250683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.250562 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v" Nov 03 08:14:23.257749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.257686 5951 generic.go:356] "Generic (PLEG): container finished" podID="0c6f0149-90e4-4644-8f93-1a5f38caff96" containerID="0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f" exitCode=0 Nov 03 08:14:23.257980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.257947 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" event={"ID":"0c6f0149-90e4-4644-8f93-1a5f38caff96","Type":"ContainerDied","Data":"0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f"} Nov 03 08:14:23.258103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.258007 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" event={"ID":"0c6f0149-90e4-4644-8f93-1a5f38caff96","Type":"ContainerDied","Data":"74a05dde1fc98a409e773c327b150cf8dfb739c06fdfdf117924f24922fee4aa"} Nov 03 08:14:23.258103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.257960 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7" Nov 03 08:14:23.267299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.265698 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" event={"ID":"aa445227-7a5e-4b11-8fd6-55fdef0cc618","Type":"ContainerDied","Data":"27c4b6579f8762831149c8f136dc5504dc8055cf1204c7fe253da1d9a87589d3"} Nov 03 08:14:23.267299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.265782 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99" Nov 03 08:14:23.267755 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.267660 5951 scope.go:122] "RemoveContainer" containerID="61a6301ddd5247d62149a43aa3964451af25105f0cfa415b7210c11b8404739c" Nov 03 08:14:23.270214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.270115 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" event={"ID":"67f93310-b389-418c-bde6-4867568cce3b","Type":"ContainerStarted","Data":"d253fe6bcf1f4ef8eb56f9daaaef50427504911f1977961a13f3c147f1ac89aa"} Nov 03 08:14:23.270214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.270184 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:23.274041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.274017 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" event={"ID":"d60e3d10-bc4e-42c9-93af-51c843e47fd9","Type":"ContainerDied","Data":"943cd52e648031d1a390fb8a4d5b696e1a10462542fa883174d6391377dc658e"} Nov 03 08:14:23.274135 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.274104 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj" Nov 03 08:14:23.279133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.279097 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp"] Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.287499 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8\" (UniqueName: \"kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8\") pod \"0c6f0149-90e4-4644-8f93-1a5f38caff96\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.287608 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca\") pod \"0c6f0149-90e4-4644-8f93-1a5f38caff96\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.287792 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs\") pod \"0c6f0149-90e4-4644-8f93-1a5f38caff96\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.287938 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca" pod "0c6f0149-90e4-4644-8f93-1a5f38caff96" (UID: "0c6f0149-90e4-4644-8f93-1a5f38caff96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.287971 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls\") pod \"0c6f0149-90e4-4644-8f93-1a5f38caff96\" (UID: \"0c6f0149-90e4-4644-8f93-1a5f38caff96\") " Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.288509 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.288522 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.288531 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c85x\" (UniqueName: \"kubernetes.io/projected/3c21cf61-4d1e-41a0-adef-173cce70680a-kube-api-access-6c85x\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.288540 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c21cf61-4d1e-41a0-adef-173cce70680a-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.288548 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6f0149-90e4-4644-8f93-1a5f38caff96-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.289491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.288683 5951 scope.go:122] "RemoveContainer" containerID="0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f" Nov 03 08:14:23.291830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.291810 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-twgqp"] Nov 03 08:14:23.299128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.299100 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs" pod "0c6f0149-90e4-4644-8f93-1a5f38caff96" (UID: "0c6f0149-90e4-4644-8f93-1a5f38caff96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:14:23.304922 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.304895 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99"] Nov 03 08:14:23.312209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.312188 5951 scope.go:122] "RemoveContainer" containerID="0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f" Nov 03 08:14:23.326858 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:23.323607 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f\": container with ID starting with 0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f not found: ID does not exist" containerID="0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f" Nov 03 08:14:23.326858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.323675 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f"} err="failed to get container status \"0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f\": rpc error: code = NotFound desc = could not find container \"0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f\": container with ID starting with 0a90e39dc5eebdfbde49e2eaa4aeb8ea58f17b3fad058a7dddccb44de104664f not found: ID does not exist" Nov 03 08:14:23.326858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.323706 5951 scope.go:122] "RemoveContainer" containerID="9ecd7b0ddadff368de7d0854e0b46d7c6b5a846757833ffa3da46bb4667cbc25" Nov 03 08:14:23.330932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.328817 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8" pod "0c6f0149-90e4-4644-8f93-1a5f38caff96" (UID: "0c6f0149-90e4-4644-8f93-1a5f38caff96"). InnerVolumeSpecName "kube-api-access-lzld8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:23.334824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.334785 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-9fb99"] Nov 03 08:14:23.338117 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.338086 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls" pod "0c6f0149-90e4-4644-8f93-1a5f38caff96" (UID: "0c6f0149-90e4-4644-8f93-1a5f38caff96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:14:23.345445 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.345415 5951 scope.go:122] "RemoveContainer" containerID="9ecd7b0ddadff368de7d0854e0b46d7c6b5a846757833ffa3da46bb4667cbc25" Nov 03 08:14:23.347538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.347523 5951 scope.go:122] "RemoveContainer" containerID="780119f888af1d1548151df330b94ee837971737b3587552f4c43f0fc067af72" Nov 03 08:14:23.347585 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.347569 5951 scope.go:122] "RemoveContainer" containerID="5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba" Nov 03 08:14:23.352980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.350480 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v"] Nov 03 08:14:23.356729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.356704 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-vrx8v"] Nov 03 08:14:23.370488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.370441 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" podStartSLOduration=3.370427395 podStartE2EDuration="3.370427395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:23.367656999 +0000 UTC m=+1202.749107915" watchObservedRunningTime="2026-11-03 08:14:23.370427395 +0000 UTC m=+1202.751878311" Nov 03 08:14:23.372041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.372016 5951 scope.go:122] "RemoveContainer" containerID="5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba" Nov 03 08:14:23.372416 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:23.372394 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba\": container with ID starting with 5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba not found: ID does not exist" containerID="5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba" Nov 03 08:14:23.372521 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:23.372419 5951 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba\": rpc error: code = NotFound desc = could not find container \"5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba\": container with ID starting with 5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba not found: ID does not exist" containerID="5ddb1dea4584a7d34d7f7ac05c76cacc783be5d26de50b3a4f747f708f53ecba" Nov 03 08:14:23.392159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.392084 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj"] Nov 03 08:14:23.392302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.392282 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.392370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.392309 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6f0149-90e4-4644-8f93-1a5f38caff96-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.392370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.392323 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzld8\" (UniqueName: \"kubernetes.io/projected/0c6f0149-90e4-4644-8f93-1a5f38caff96-kube-api-access-lzld8\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:23.398802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.398757 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-mfssj"] Nov 03 08:14:23.429692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.429663 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl"] Nov 03 08:14:23.432227 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:23.432195 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae3a0c9_d75e_4b8a_9552_e49ea13d9737.slice/crio-9a2093b61a6c9232569f61848a204aad5ff9a49abdb8d7b54f4da663334be52a WatchSource:0}: Error finding container 9a2093b61a6c9232569f61848a204aad5ff9a49abdb8d7b54f4da663334be52a: Status 404 returned error can't find the container with id 9a2093b61a6c9232569f61848a204aad5ff9a49abdb8d7b54f4da663334be52a Nov 03 08:14:23.593353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.593273 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7"] Nov 03 08:14:23.611605 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:23.611565 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e255cbee-d9d75984-2cfv7"] Nov 03 08:14:24.283740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.283699 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" event={"ID":"fae3a0c9-d75e-4b8a-9552-e49ea13d9737","Type":"ContainerStarted","Data":"4c6b9e50e17ee737f64987604d95d70c7d9bfa992f9b40cf008167c390e8106d"} Nov 03 08:14:24.284249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.283765 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:24.284249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.283782 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" event={"ID":"fae3a0c9-d75e-4b8a-9552-e49ea13d9737","Type":"ContainerStarted","Data":"9a2093b61a6c9232569f61848a204aad5ff9a49abdb8d7b54f4da663334be52a"} Nov 03 08:14:24.285542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.285513 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" event={"ID":"67f93310-b389-418c-bde6-4867568cce3b","Type":"ContainerStarted","Data":"a37d9ed55ca38d585334c5bdc95b74f8550f34daa37347303cd07aa1ae70320d"} Nov 03 08:14:24.303322 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.303280 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" podStartSLOduration=3.30326581 podStartE2EDuration="3.30326581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:24.301510636 +0000 UTC m=+1203.682961550" watchObservedRunningTime="2026-11-03 08:14:24.30326581 +0000 UTC m=+1203.684717106" Nov 03 08:14:24.741830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.741738 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6f0149-90e4-4644-8f93-1a5f38caff96" path="/var/lib/kubelet/pods/0c6f0149-90e4-4644-8f93-1a5f38caff96/volumes" Nov 03 08:14:24.742339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.742318 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" path="/var/lib/kubelet/pods/2a52f05e-2638-4c97-ae7e-438d5a93382d/volumes" Nov 03 08:14:24.742857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.742839 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" path="/var/lib/kubelet/pods/3c21cf61-4d1e-41a0-adef-173cce70680a/volumes" Nov 03 08:14:24.743758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.743737 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" path="/var/lib/kubelet/pods/aa445227-7a5e-4b11-8fd6-55fdef0cc618/volumes" Nov 03 08:14:24.744179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:24.744166 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" path="/var/lib/kubelet/pods/d60e3d10-bc4e-42c9-93af-51c843e47fd9/volumes" Nov 03 08:14:25.737673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.737599 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28"] Nov 03 08:14:25.738857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738824 5951 cpu_manager.go:414] "containerMap: removing container" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" Nov 03 08:14:25.738857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738859 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" Nov 03 08:14:25.738981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738883 5951 cpu_manager.go:414] "containerMap: removing container" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" Nov 03 08:14:25.738981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738890 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" Nov 03 08:14:25.738981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738949 5951 cpu_manager.go:414] "containerMap: removing container" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" Nov 03 08:14:25.738981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738958 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" Nov 03 08:14:25.738981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738980 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" Nov 03 08:14:25.739129 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.738987 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" Nov 03 08:14:25.739129 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739007 5951 cpu_manager.go:414] "containerMap: removing container" podUID="0c6f0149-90e4-4644-8f93-1a5f38caff96" containerName="maas-api" Nov 03 08:14:25.739129 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739014 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6f0149-90e4-4644-8f93-1a5f38caff96" containerName="maas-api" Nov 03 08:14:25.739215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739180 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa445227-7a5e-4b11-8fd6-55fdef0cc618" containerName="istio-proxy" Nov 03 08:14:25.739215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739197 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a52f05e-2638-4c97-ae7e-438d5a93382d" containerName="istio-proxy" Nov 03 08:14:25.739215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739209 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6f0149-90e4-4644-8f93-1a5f38caff96" containerName="maas-api" Nov 03 08:14:25.739320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739221 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c21cf61-4d1e-41a0-adef-173cce70680a" containerName="istio-proxy" Nov 03 08:14:25.739320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.739230 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60e3d10-bc4e-42c9-93af-51c843e47fd9" containerName="istio-proxy" Nov 03 08:14:25.745270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.745244 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.748729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.748192 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-228695-serving-cert\"" Nov 03 08:14:25.752830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.752800 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28"] Nov 03 08:14:25.842757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.842710 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.842929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.842863 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzjs\" (UniqueName: \"kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.843063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.843047 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.843176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.843160 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.945037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.944992 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzjs\" (UniqueName: \"kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.945244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.945140 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.945244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.945222 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.945367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.945354 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.946197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.946164 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.948771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.948737 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.948908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.948882 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:25.953283 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:25.953258 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzjs\" (UniqueName: \"kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs\") pod \"maas-api-e2e-api-b-228695-65c7dcfbb6-24g28\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:26.069539 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:26.069508 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:26.387239 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:26.385918 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28"] Nov 03 08:14:26.536457 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:26.532803 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k"] Nov 03 08:14:26.536457 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:26.533035 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" podUID="ae7d95e1-a48b-4787-a517-cf8c0c080916" containerName="maas-api" containerID="cri-o://82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50" gracePeriod=30 Nov 03 08:14:27.186162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.186130 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:14:27.331395 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.331286 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca\") pod \"ae7d95e1-a48b-4787-a517-cf8c0c080916\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " Nov 03 08:14:27.331563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.331444 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2\" (UniqueName: \"kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2\") pod \"ae7d95e1-a48b-4787-a517-cf8c0c080916\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " Nov 03 08:14:27.332134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.331694 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs\") pod \"ae7d95e1-a48b-4787-a517-cf8c0c080916\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " Nov 03 08:14:27.332134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.331769 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls\") pod \"ae7d95e1-a48b-4787-a517-cf8c0c080916\" (UID: \"ae7d95e1-a48b-4787-a517-cf8c0c080916\") " Nov 03 08:14:27.334983 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.334926 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca" pod "ae7d95e1-a48b-4787-a517-cf8c0c080916" (UID: "ae7d95e1-a48b-4787-a517-cf8c0c080916"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:27.343516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.343481 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs" pod "ae7d95e1-a48b-4787-a517-cf8c0c080916" (UID: "ae7d95e1-a48b-4787-a517-cf8c0c080916"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:14:27.344602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.344573 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2" pod "ae7d95e1-a48b-4787-a517-cf8c0c080916" (UID: "ae7d95e1-a48b-4787-a517-cf8c0c080916"). InnerVolumeSpecName "kube-api-access-fhmr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:27.346928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.346793 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls" pod "ae7d95e1-a48b-4787-a517-cf8c0c080916" (UID: "ae7d95e1-a48b-4787-a517-cf8c0c080916"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:14:27.356722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.351118 5951 generic.go:356] "Generic (PLEG): container finished" podID="ae7d95e1-a48b-4787-a517-cf8c0c080916" containerID="82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50" exitCode=0 Nov 03 08:14:27.356722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.351208 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" event={"ID":"ae7d95e1-a48b-4787-a517-cf8c0c080916","Type":"ContainerDied","Data":"82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50"} Nov 03 08:14:27.356722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.351258 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" event={"ID":"ae7d95e1-a48b-4787-a517-cf8c0c080916","Type":"ContainerDied","Data":"7241539f0c3b64dbf59d080d5f51fa01793e84ed554ab7e9c25849f4cbdd9c6e"} Nov 03 08:14:27.356722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.351287 5951 scope.go:122] "RemoveContainer" containerID="82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50" Nov 03 08:14:27.356722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.351420 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k" Nov 03 08:14:27.363387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.363001 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" event={"ID":"ae1c5f9e-1482-4821-9805-ce721e782b7d","Type":"ContainerStarted","Data":"32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa"} Nov 03 08:14:27.363387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.363059 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" event={"ID":"ae1c5f9e-1482-4821-9805-ce721e782b7d","Type":"ContainerStarted","Data":"78173504f8dcc6f947fe588d971bb257f723271e57417eb1e69f4698f6f873f2"} Nov 03 08:14:27.363387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.363328 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:27.375983 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.375959 5951 scope.go:122] "RemoveContainer" containerID="82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50" Nov 03 08:14:27.376335 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:27.376305 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50\": container with ID starting with 82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50 not found: ID does not exist" containerID="82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50" Nov 03 08:14:27.376426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.376345 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50"} err="failed to get container status \"82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50\": rpc error: code = NotFound desc = could not find container \"82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50\": container with ID starting with 82c489ee9badac7b329d254a6e429dc829c70ff7e38fa38d0d40b68ed3e65a50 not found: ID does not exist" Nov 03 08:14:27.386454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.386407 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" podStartSLOduration=1.816897978 podStartE2EDuration="2.386392603s" totalImagesPullingTime="569.494625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:27.378753536 +0000 UTC m=+1206.760204467" watchObservedRunningTime="2026-11-03 08:14:27.386392603 +0000 UTC m=+1206.767843520" Nov 03 08:14:27.419031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.414153 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k"] Nov 03 08:14:27.419031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.414218 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb887217-7f768c896b-7c85k"] Nov 03 08:14:27.434935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.434897 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:27.434935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.434935 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7d95e1-a48b-4787-a517-cf8c0c080916-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:27.435216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.434953 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7d95e1-a48b-4787-a517-cf8c0c080916-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:27.435216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:27.434967 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmr2\" (UniqueName: \"kubernetes.io/projected/ae7d95e1-a48b-4787-a517-cf8c0c080916-kube-api-access-fhmr2\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:28.742516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:28.742484 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7d95e1-a48b-4787-a517-cf8c0c080916" path="/var/lib/kubelet/pods/ae7d95e1-a48b-4787-a517-cf8c0c080916/volumes" Nov 03 08:14:31.581916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.581880 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4"] Nov 03 08:14:31.583228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.583200 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ae7d95e1-a48b-4787-a517-cf8c0c080916" containerName="maas-api" Nov 03 08:14:31.583228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.583228 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7d95e1-a48b-4787-a517-cf8c0c080916" containerName="maas-api" Nov 03 08:14:31.583443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.583429 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7d95e1-a48b-4787-a517-cf8c0c080916" containerName="maas-api" Nov 03 08:14:31.589554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.589530 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.600963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.600893 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-950e06-openshift-default-dockercfg-rz8ws\"" Nov 03 08:14:31.631913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.631761 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4"] Nov 03 08:14:31.666263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666187 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666327 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666385 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666412 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666459 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666590 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666680 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666736 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqg9h\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.666812 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.666809 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769300 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769510 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769395 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769510 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769424 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769510 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769472 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769720 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769811 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.769925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769874 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqg9h\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.770176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.769951 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.770176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.770052 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.770380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.770353 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.772954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.772929 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.776781 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.776757 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.776913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.776888 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.777339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.777204 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.777339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.777299 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.777489 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.777457 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.779254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.779232 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.780128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.780011 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqg9h\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h\") pod \"e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:31.967749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:31.958937 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:32.057952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.057873 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6"] Nov 03 08:14:32.072763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.058196 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" containerID="cri-o://26103917c5b748bd59c3b1ce3721d1dce12633e807a0ee46b2876a1531dcd2a0" gracePeriod=30 Nov 03 08:14:32.089420 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.089384 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq"] Nov 03 08:14:32.089630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.089603 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" containerID="cri-o://44b82842792bc458d877214cc73f3e0af1a24a2906034460e5b4dac6a86dac90" gracePeriod=30 Nov 03 08:14:32.209804 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:14:32.209761 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:14:32.500381 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:32.500347 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e71c051_5971_42b9_80a7_6b85964c8350.slice/crio-f1a2bb2076fc3586fe2b613831e5361971eb20a23f0368ff64249138e6a32938 WatchSource:0}: Error finding container f1a2bb2076fc3586fe2b613831e5361971eb20a23f0368ff64249138e6a32938: Status 404 returned error can't find the container with id f1a2bb2076fc3586fe2b613831e5361971eb20a23f0368ff64249138e6a32938 Nov 03 08:14:32.501852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.501825 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4"] Nov 03 08:14:32.504274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.504226 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:32.504372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.504311 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:32.504372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:32.504355 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:14:33.382507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:33.382479 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:14:33.494604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:33.494563 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" event={"ID":"8e71c051-5971-42b9-80a7-6b85964c8350","Type":"ContainerStarted","Data":"d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067"} Nov 03 08:14:33.494805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:33.494630 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" event={"ID":"8e71c051-5971-42b9-80a7-6b85964c8350","Type":"ContainerStarted","Data":"f1a2bb2076fc3586fe2b613831e5361971eb20a23f0368ff64249138e6a32938"} Nov 03 08:14:33.534726 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:33.534666 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podStartSLOduration=2.534628338 podStartE2EDuration="2.534628338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:33.525180133 +0000 UTC m=+1212.906631075" watchObservedRunningTime="2026-11-03 08:14:33.534628338 +0000 UTC m=+1212.916079258" Nov 03 08:14:33.960466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:33.960430 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:33.967840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:33.967811 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:34.519817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:34.519782 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:34.522758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:34.522734 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:14:35.302299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:35.302268 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:14:35.302625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:35.302606 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:14:37.200096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.200061 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w"] Nov 03 08:14:37.206315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.206272 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.212934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.212852 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-950e06\"" Nov 03 08:14:37.215919 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.213300 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-950e06-dockercfg-gw2vx\"" Nov 03 08:14:37.222234 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.222202 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w"] Nov 03 08:14:37.263034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.262995 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg"] Nov 03 08:14:37.269268 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.269243 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.287853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.286785 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg"] Nov 03 08:14:37.311503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.311466 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jrw\" (UniqueName: \"kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw\") pod \"payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.312267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.312200 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.628214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.628178 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.628525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.628501 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jrw\" (UniqueName: \"kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw\") pod \"payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.628712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.628693 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbwt\" (UniqueName: \"kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt\") pod \"payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.628960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.628944 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.640485 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.640450 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.663981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.663881 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jrw\" (UniqueName: \"kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw\") pod \"payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.702764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.686484 5951 generic.go:356] "Generic (PLEG): container finished" podID="a974d7b3-dd54-4006-b009-736db4c87ece" containerID="44b82842792bc458d877214cc73f3e0af1a24a2906034460e5b4dac6a86dac90" exitCode=0 Nov 03 08:14:37.702764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.686516 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" event={"ID":"a974d7b3-dd54-4006-b009-736db4c87ece","Type":"ContainerDied","Data":"44b82842792bc458d877214cc73f3e0af1a24a2906034460e5b4dac6a86dac90"} Nov 03 08:14:37.702764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.692241 5951 generic.go:356] "Generic (PLEG): container finished" podID="043d6408-4f88-456b-a6e8-524a42188208" containerID="26103917c5b748bd59c3b1ce3721d1dce12633e807a0ee46b2876a1531dcd2a0" exitCode=0 Nov 03 08:14:37.702764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.692382 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" event={"ID":"043d6408-4f88-456b-a6e8-524a42188208","Type":"ContainerDied","Data":"26103917c5b748bd59c3b1ce3721d1dce12633e807a0ee46b2876a1531dcd2a0"} Nov 03 08:14:37.772297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.761557 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.772297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.761709 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbwt\" (UniqueName: \"kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt\") pod \"payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.772297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.763110 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.795163 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.790805 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbwt\" (UniqueName: \"kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt\") pod \"payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.836013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.835979 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:14:37.864688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.864616 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:14:37.865706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.865674 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:37.885348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.885316 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp"] Nov 03 08:14:37.886387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.886363 5951 cpu_manager.go:414] "containerMap: removing container" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" Nov 03 08:14:37.886387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.886389 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" Nov 03 08:14:37.886533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.886457 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" Nov 03 08:14:37.886533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.886467 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" Nov 03 08:14:37.886691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.886661 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" containerName="istio-proxy" Nov 03 08:14:37.886691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.886689 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="043d6408-4f88-456b-a6e8-524a42188208" containerName="istio-proxy" Nov 03 08:14:37.891073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.891048 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:37.898864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.898613 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:37.908521 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.908496 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ba69a0-serving-cert\"" Nov 03 08:14:37.919981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.919902 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp"] Nov 03 08:14:37.964501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964464 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.964725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964518 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.964725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964565 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.964725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964593 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.964725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964625 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.964725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964699 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.965866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964756 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.965866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.964784 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.968367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.968034 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.968367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.968104 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.968367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.968154 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.968367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.968204 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.968873 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.968961 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.969036 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.969079 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.969161 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy\") pod \"a974d7b3-dd54-4006-b009-736db4c87ece\" (UID: \"a974d7b3-dd54-4006-b009-736db4c87ece\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.969223 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data\") pod \"043d6408-4f88-456b-a6e8-524a42188208\" (UID: \"043d6408-4f88-456b-a6e8-524a42188208\") " Nov 03 08:14:37.973701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.969596 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:37.976102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.974798 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjn7\" (UniqueName: \"kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:37.976102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.974855 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:37.976102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.975201 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:37.976544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.976321 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.976907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.976721 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.976907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.976777 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.977200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.977027 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.980558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.979405 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.980558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.979511 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:37.980558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.979745 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.980558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.979768 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.980558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.980083 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:37.980558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.980158 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.985010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.984467 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:14:37.987573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.986540 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.987573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.987092 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:37.988315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.988277 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:14:37.988616 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.988576 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "kube-api-access-7k24n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:37.992694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.992070 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy" pod "043d6408-4f88-456b-a6e8-524a42188208" (UID: "043d6408-4f88-456b-a6e8-524a42188208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:14:37.992694 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.992358 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "kube-api-access-4cnj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:37.993250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:37.993223 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token" pod "a974d7b3-dd54-4006-b009-736db4c87ece" (UID: "a974d7b3-dd54-4006-b009-736db4c87ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:14:38.076884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.076845 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.077124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.076923 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.077124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077012 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjn7\" (UniqueName: \"kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.077124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077054 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077159 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043d6408-4f88-456b-a6e8-524a42188208-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077177 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077190 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnj5\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-kube-api-access-4cnj5\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077205 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077219 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a974d7b3-dd54-4006-b009-736db4c87ece-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077231 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077245 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a974d7b3-dd54-4006-b009-736db4c87ece-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077262 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077275 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077288 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077302 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077316 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a974d7b3-dd54-4006-b009-736db4c87ece-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077330 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k24n\" (UniqueName: \"kubernetes.io/projected/043d6408-4f88-456b-a6e8-524a42188208-kube-api-access-7k24n\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077344 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043d6408-4f88-456b-a6e8-524a42188208-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077358 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.077361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077371 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.078342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077383 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a974d7b3-dd54-4006-b009-736db4c87ece-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.078342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.077397 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043d6408-4f88-456b-a6e8-524a42188208-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:14:38.078342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.078038 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.080530 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.080504 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.080630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.080504 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.088181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.088158 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjn7\" (UniqueName: \"kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7\") pod \"maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.222448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.221761 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w"] Nov 03 08:14:38.228612 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:38.228576 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666fb4d8_b2dd_43ad_8df2_c3821bbc15ad.slice/crio-a87f2a3df50b0c5a44fb1ea39362118483d9673c726a7a8aea680df1819cdee9 WatchSource:0}: Error finding container a87f2a3df50b0c5a44fb1ea39362118483d9673c726a7a8aea680df1819cdee9: Status 404 returned error can't find the container with id a87f2a3df50b0c5a44fb1ea39362118483d9673c726a7a8aea680df1819cdee9 Nov 03 08:14:38.231051 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.230179 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:38.264254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.264228 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg"] Nov 03 08:14:38.613410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.613373 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp"] Nov 03 08:14:38.617873 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:38.617844 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a9e81b_9bc9_4de9_8e7c_98ecf34b8575.slice/crio-e37bc4695714217dfbc11935058a1bbea982328b7274a6185af38572cf564db4 WatchSource:0}: Error finding container e37bc4695714217dfbc11935058a1bbea982328b7274a6185af38572cf564db4: Status 404 returned error can't find the container with id e37bc4695714217dfbc11935058a1bbea982328b7274a6185af38572cf564db4 Nov 03 08:14:38.715598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.710383 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" event={"ID":"a974d7b3-dd54-4006-b009-736db4c87ece","Type":"ContainerDied","Data":"a77f905f8734ebca175e9d3c07124b15b64274b5530f39a87ba5b11706850bf6"} Nov 03 08:14:38.715598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.710460 5951 scope.go:122] "RemoveContainer" containerID="44b82842792bc458d877214cc73f3e0af1a24a2906034460e5b4dac6a86dac90" Nov 03 08:14:38.715598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.710630 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq" Nov 03 08:14:38.727272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.727228 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" event={"ID":"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9","Type":"ContainerStarted","Data":"ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502"} Nov 03 08:14:38.727446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.727299 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" event={"ID":"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9","Type":"ContainerStarted","Data":"0e40ebd4dbf3cc5bcde90d41cc17e9990fc0f6d5f5d01a6f2be9baa8e6546ab9"} Nov 03 08:14:38.728994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.728968 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:38.736925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.736902 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" Nov 03 08:14:38.750052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.749588 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" podStartSLOduration=1.749571677 podStartE2EDuration="1.749571677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:38.74455185 +0000 UTC m=+1218.126002770" watchObservedRunningTime="2026-11-03 08:14:38.749571677 +0000 UTC m=+1218.131022595" Nov 03 08:14:38.754004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.753965 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:38.754140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.754011 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" event={"ID":"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad","Type":"ContainerStarted","Data":"9e4348fb79ded1f4580966c693fdc992c5aaa449928e0b054f7f49de01d68532"} Nov 03 08:14:38.754140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.754039 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" event={"ID":"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad","Type":"ContainerStarted","Data":"a87f2a3df50b0c5a44fb1ea39362118483d9673c726a7a8aea680df1819cdee9"} Nov 03 08:14:38.754140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.754060 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6" event={"ID":"043d6408-4f88-456b-a6e8-524a42188208","Type":"ContainerDied","Data":"cd186d2f1a3044a744031ee4e32598862e2f67979804689d3ceb0f360c503117"} Nov 03 08:14:38.754140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.754076 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" event={"ID":"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575","Type":"ContainerStarted","Data":"e37bc4695714217dfbc11935058a1bbea982328b7274a6185af38572cf564db4"} Nov 03 08:14:38.754140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.754104 5951 scope.go:122] "RemoveContainer" containerID="26103917c5b748bd59c3b1ce3721d1dce12633e807a0ee46b2876a1531dcd2a0" Nov 03 08:14:38.791886 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.791799 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" podStartSLOduration=1.791781106 podStartE2EDuration="1.791781106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:38.774821136 +0000 UTC m=+1218.156272079" watchObservedRunningTime="2026-11-03 08:14:38.791781106 +0000 UTC m=+1218.173232024" Nov 03 08:14:38.793468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.793445 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq"] Nov 03 08:14:38.800817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.800789 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-2ccnq"] Nov 03 08:14:38.809469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.809441 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6"] Nov 03 08:14:38.816741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:38.816711 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-6ggx6"] Nov 03 08:14:39.805296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:39.805167 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" event={"ID":"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575","Type":"ContainerStarted","Data":"f37b1a6a9868ac205d33a16f8de2ec940a1c4b2fe2610f5cabd3f282e89da9ae"} Nov 03 08:14:39.825411 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:39.825355 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" podStartSLOduration=2.393486567 podStartE2EDuration="2.825336608s" totalImagesPullingTime="431.850041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:39.821088371 +0000 UTC m=+1219.202539287" watchObservedRunningTime="2026-11-03 08:14:39.825336608 +0000 UTC m=+1219.206787519" Nov 03 08:14:40.740946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:40.740908 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043d6408-4f88-456b-a6e8-524a42188208" path="/var/lib/kubelet/pods/043d6408-4f88-456b-a6e8-524a42188208/volumes" Nov 03 08:14:40.741582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:40.741563 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a974d7b3-dd54-4006-b009-736db4c87ece" path="/var/lib/kubelet/pods/a974d7b3-dd54-4006-b009-736db4c87ece/volumes" Nov 03 08:14:40.814614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:40.814581 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:46.831824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:46.831791 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:14:49.538013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.537973 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp"] Nov 03 08:14:49.546331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.546137 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:49.548631 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.548602 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp"] Nov 03 08:14:49.550260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.549908 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9b559e\"" Nov 03 08:14:49.580379 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.580344 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4"] Nov 03 08:14:49.585534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.585505 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.600326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.597451 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4"] Nov 03 08:14:49.605179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.605032 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8ln\" (UniqueName: \"kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln\") pod \"payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:49.605179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.605173 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdjx\" (UniqueName: \"kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx\") pod \"payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.605402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.605231 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:49.605402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.605352 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.634668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.630479 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9b559e-dockercfg-5qxz7\"" Nov 03 08:14:49.749763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.728887 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.749763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.729074 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8ln\" (UniqueName: \"kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln\") pod \"payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:49.749763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.729226 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdjx\" (UniqueName: \"kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx\") pod \"payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.749763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.729288 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:49.759679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.750369 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:49.759679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.751593 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.938790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.918385 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdjx\" (UniqueName: \"kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx\") pod \"payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:49.939026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.937971 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:14:49.945100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:49.945008 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8ln\" (UniqueName: \"kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln\") pod \"payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:50.174231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.174201 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:50.209695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.209533 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:50.508627 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:50.508597 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9415064_28a2_41a2_bfb8_0a06634ab720.slice/crio-2688215e04a9f1a5a1a284ec23bdd979a06dcce40f0d6b6c66443cafd420773b WatchSource:0}: Error finding container 2688215e04a9f1a5a1a284ec23bdd979a06dcce40f0d6b6c66443cafd420773b: Status 404 returned error can't find the container with id 2688215e04a9f1a5a1a284ec23bdd979a06dcce40f0d6b6c66443cafd420773b Nov 03 08:14:50.512561 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.512537 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp"] Nov 03 08:14:50.555445 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:14:50.555416 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985bd2be_13e7_40a7_9022_38979241bc88.slice/crio-aeac0b4acc782eb9b0b0832b96d7629a09d4977db076428185933de3b7d1fd4e WatchSource:0}: Error finding container aeac0b4acc782eb9b0b0832b96d7629a09d4977db076428185933de3b7d1fd4e: Status 404 returned error can't find the container with id aeac0b4acc782eb9b0b0832b96d7629a09d4977db076428185933de3b7d1fd4e Nov 03 08:14:50.557166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.557134 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4"] Nov 03 08:14:50.818272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.818228 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:14:50.950026 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.949979 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" event={"ID":"985bd2be-13e7-40a7-9022-38979241bc88","Type":"ContainerStarted","Data":"feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913"} Nov 03 08:14:50.950230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.950064 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:14:50.950230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.950088 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" event={"ID":"985bd2be-13e7-40a7-9022-38979241bc88","Type":"ContainerStarted","Data":"aeac0b4acc782eb9b0b0832b96d7629a09d4977db076428185933de3b7d1fd4e"} Nov 03 08:14:50.955900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.955866 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" event={"ID":"e9415064-28a2-41a2-bfb8-0a06634ab720","Type":"ContainerStarted","Data":"f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af"} Nov 03 08:14:50.956100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.955926 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" event={"ID":"e9415064-28a2-41a2-bfb8-0a06634ab720","Type":"ContainerStarted","Data":"2688215e04a9f1a5a1a284ec23bdd979a06dcce40f0d6b6c66443cafd420773b"} Nov 03 08:14:50.956100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.955958 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:14:50.970990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.970940 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" podStartSLOduration=1.9709249340000001 podStartE2EDuration="1.970924934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:50.968431368 +0000 UTC m=+1230.349882285" watchObservedRunningTime="2026-11-03 08:14:50.970924934 +0000 UTC m=+1230.352375867" Nov 03 08:14:50.987997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:50.987806 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" podStartSLOduration=1.9877865670000001 podStartE2EDuration="1.987786567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:50.984701988 +0000 UTC m=+1230.366152932" watchObservedRunningTime="2026-11-03 08:14:50.987786567 +0000 UTC m=+1230.369237485" Nov 03 08:14:59.172388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:14:59.172351 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp"] Nov 03 08:15:00.355978 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.355728 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf"] Nov 03 08:15:00.391358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.385525 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx"] Nov 03 08:15:00.391358 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.387115 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.401256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.401222 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf"] Nov 03 08:15:00.401429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.401285 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx"] Nov 03 08:15:00.401429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.401367 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.468099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.468059 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.468299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.468184 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpwr\" (UniqueName: \"kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr\") pod \"maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.468299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.468267 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.468438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.468313 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257ct\" (UniqueName: \"kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct\") pod \"maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.478104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.478077 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\"" Nov 03 08:15:00.570297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.570262 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpwr\" (UniqueName: \"kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr\") pod \"maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.570470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.570376 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.570470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.570442 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257ct\" (UniqueName: \"kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct\") pod \"maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.570686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.570625 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.571065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.571042 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.571327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.571303 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.581794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.581774 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257ct\" (UniqueName: \"kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct\") pod \"maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.583320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.583303 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpwr\" (UniqueName: \"kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr\") pod \"maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.718444 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.718358 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:00.740227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.738857 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:00.992305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.992227 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4"] Nov 03 08:15:00.998250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:00.998228 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.015868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.015583 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-179c16-openshift-default-dockercfg-5lffm\"" Nov 03 08:15:01.025485 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.025457 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4"] Nov 03 08:15:01.060890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.060847 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt"] Nov 03 08:15:01.072771 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:15:01.072728 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60aafb5d_1519_4aa7_8135_01905662958b.slice/crio-fd5fc135d506333c2e762cd62ed3759fa15e9c67c54741ce4931fada68b393d4 WatchSource:0}: Error finding container fd5fc135d506333c2e762cd62ed3759fa15e9c67c54741ce4931fada68b393d4: Status 404 returned error can't find the container with id fd5fc135d506333c2e762cd62ed3759fa15e9c67c54741ce4931fada68b393d4 Nov 03 08:15:01.076351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.076326 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt"] Nov 03 08:15:01.076458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.076372 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf"] Nov 03 08:15:01.076519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.076465 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.090136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.090068 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e0f576\"" Nov 03 08:15:01.094020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.093533 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094058 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094171 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvrc\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094332 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094421 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094473 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094632 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094693 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.094790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.094761 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.109991 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.109964 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e0f576-dockercfg-8tn7p\"" Nov 03 08:15:01.158534 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:15:01.158499 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0effc8_50e2_4ea6_86be_0649ad5aab23.slice/crio-2c05181cff5bc5bcdd2558b4c79593cc04c9583597cf424efea9c446286af3af WatchSource:0}: Error finding container 2c05181cff5bc5bcdd2558b4c79593cc04c9583597cf424efea9c446286af3af: Status 404 returned error can't find the container with id 2c05181cff5bc5bcdd2558b4c79593cc04c9583597cf424efea9c446286af3af Nov 03 08:15:01.161748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.161717 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx"] Nov 03 08:15:01.197850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.197761 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.197850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.197823 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.197867 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.198004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.197968 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.197999 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198163 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198063 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198224 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198215 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198246 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198300 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqr26\" (UniqueName: \"kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26\") pod \"payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.198345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198333 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvrc\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198481 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.198869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.198801 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.200050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.200019 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.201063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.201032 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.201936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.201914 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.204321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.204300 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.211423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.211395 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.212240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.212208 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.212478 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.212453 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.217266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.216973 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvrc\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc\") pod \"e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.313697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.312250 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.313697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.312544 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqr26\" (UniqueName: \"kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26\") pod \"payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.314497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.314446 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.330849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.330780 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqr26\" (UniqueName: \"kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26\") pod \"payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.337342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.337035 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:01.472696 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.418069 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:01.969196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.969026 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:15:01.969196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.969109 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:15:01.986830 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:15:01.986799 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4853d6b_f6ba_4245_89aa_2069c46cea2a.slice/crio-87251473dc7575dcfdb6fa1c66dcd157cf6452fbeb39b6dccb4bf64456f78912 WatchSource:0}: Error finding container 87251473dc7575dcfdb6fa1c66dcd157cf6452fbeb39b6dccb4bf64456f78912: Status 404 returned error can't find the container with id 87251473dc7575dcfdb6fa1c66dcd157cf6452fbeb39b6dccb4bf64456f78912 Nov 03 08:15:01.987143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.987120 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4"] Nov 03 08:15:01.994730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.990551 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:15:01.994730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.990661 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:15:01.994730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:01.990706 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:15:02.006437 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:02.006406 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:15:02.044588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.044531 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt"] Nov 03 08:15:02.127288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.127250 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" event={"ID":"a4853d6b-f6ba-4245-89aa-2069c46cea2a","Type":"ContainerStarted","Data":"87251473dc7575dcfdb6fa1c66dcd157cf6452fbeb39b6dccb4bf64456f78912"} Nov 03 08:15:02.130174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.129797 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerStarted","Data":"a2a596a18fbdb9d6f129e77a8fc70305c84132ba037db83b95e56a759ac9d5c6"} Nov 03 08:15:02.130174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.129859 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerStarted","Data":"2c05181cff5bc5bcdd2558b4c79593cc04c9583597cf424efea9c446286af3af"} Nov 03 08:15:02.136147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.135182 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerStarted","Data":"69391bd1824a24ff82fe05c3363c707ee7005d24b918df35852f26bbd68705d3"} Nov 03 08:15:02.136147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.135245 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerStarted","Data":"fd5fc135d506333c2e762cd62ed3759fa15e9c67c54741ce4931fada68b393d4"} Nov 03 08:15:02.162632 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.162514 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" podStartSLOduration=2.162497192 podStartE2EDuration="2.162497192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.151484324 +0000 UTC m=+1241.532935242" watchObservedRunningTime="2026-11-03 08:15:02.162497192 +0000 UTC m=+1241.543948109" Nov 03 08:15:02.169772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.169680 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" event={"ID":"5bb7caeb-48b0-4683-9dc2-08d96a124b64","Type":"ContainerStarted","Data":"5ec97a962f2b2189bd8c263ae8d282f5b9b4a8261fc8dc00b7e202b40ecd1aa9"} Nov 03 08:15:02.177732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:02.175427 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" podStartSLOduration=2.175409881 podStartE2EDuration="2.175409881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.171696166 +0000 UTC m=+1241.553147082" watchObservedRunningTime="2026-11-03 08:15:02.175409881 +0000 UTC m=+1241.556860800" Nov 03 08:15:03.180283 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.180248 5951 generic.go:356] "Generic (PLEG): container finished" podID="60aafb5d-1519-4aa7-8135-01905662958b" containerID="69391bd1824a24ff82fe05c3363c707ee7005d24b918df35852f26bbd68705d3" exitCode=60 Nov 03 08:15:03.180876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.180432 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerDied","Data":"69391bd1824a24ff82fe05c3363c707ee7005d24b918df35852f26bbd68705d3"} Nov 03 08:15:03.180876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.180873 5951 scope.go:122] "RemoveContainer" containerID="69391bd1824a24ff82fe05c3363c707ee7005d24b918df35852f26bbd68705d3" Nov 03 08:15:03.182300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.182269 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" event={"ID":"5bb7caeb-48b0-4683-9dc2-08d96a124b64","Type":"ContainerStarted","Data":"3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc"} Nov 03 08:15:03.182406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.182341 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:03.185282 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.185257 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" event={"ID":"a4853d6b-f6ba-4245-89aa-2069c46cea2a","Type":"ContainerStarted","Data":"791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e"} Nov 03 08:15:03.188227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.188201 5951 generic.go:356] "Generic (PLEG): container finished" podID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerID="a2a596a18fbdb9d6f129e77a8fc70305c84132ba037db83b95e56a759ac9d5c6" exitCode=60 Nov 03 08:15:03.188321 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.188281 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerDied","Data":"a2a596a18fbdb9d6f129e77a8fc70305c84132ba037db83b95e56a759ac9d5c6"} Nov 03 08:15:03.188496 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.188481 5951 scope.go:122] "RemoveContainer" containerID="a2a596a18fbdb9d6f129e77a8fc70305c84132ba037db83b95e56a759ac9d5c6" Nov 03 08:15:03.268166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.268108 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podStartSLOduration=3.268094073 podStartE2EDuration="3.268094073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.265091344 +0000 UTC m=+1242.646542260" watchObservedRunningTime="2026-11-03 08:15:03.268094073 +0000 UTC m=+1242.649544989" Nov 03 08:15:03.291027 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.290964 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" podStartSLOduration=2.290947959 podStartE2EDuration="2.290947959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.285854707 +0000 UTC m=+1242.667305625" watchObservedRunningTime="2026-11-03 08:15:03.290947959 +0000 UTC m=+1242.672398877" Nov 03 08:15:03.337576 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.337543 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:03.343727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:03.343682 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:04.174074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.174029 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" podUID="158efa57-2705-48b6-a778-f7afecc9fde5" containerName="payload-processing" containerID="cri-o://8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407" gracePeriod=25 Nov 03 08:15:04.204039 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.204005 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerStarted","Data":"9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061"} Nov 03 08:15:04.208088 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.207353 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerStarted","Data":"74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3"} Nov 03 08:15:04.208088 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.207412 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:04.208520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.208502 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:15:04.598066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.598037 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:15:04.760497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.760388 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume\") pod \"158efa57-2705-48b6-a778-f7afecc9fde5\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " Nov 03 08:15:04.761033 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.761004 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l\" (UniqueName: \"kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l\") pod \"158efa57-2705-48b6-a778-f7afecc9fde5\" (UID: \"158efa57-2705-48b6-a778-f7afecc9fde5\") " Nov 03 08:15:04.762003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.761504 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume" pod "158efa57-2705-48b6-a778-f7afecc9fde5" (UID: "158efa57-2705-48b6-a778-f7afecc9fde5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:04.762479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.762441 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158efa57-2705-48b6-a778-f7afecc9fde5-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:04.786977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.785400 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l" pod "158efa57-2705-48b6-a778-f7afecc9fde5" (UID: "158efa57-2705-48b6-a778-f7afecc9fde5"). InnerVolumeSpecName "kube-api-access-mvf6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:04.864228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:04.864191 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvf6l\" (UniqueName: \"kubernetes.io/projected/158efa57-2705-48b6-a778-f7afecc9fde5-kube-api-access-mvf6l\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:05.221377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.221343 5951 generic.go:356] "Generic (PLEG): container finished" podID="158efa57-2705-48b6-a778-f7afecc9fde5" containerID="8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407" exitCode=0 Nov 03 08:15:05.221915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.221409 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" Nov 03 08:15:05.221915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.221428 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" event={"ID":"158efa57-2705-48b6-a778-f7afecc9fde5","Type":"ContainerDied","Data":"8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407"} Nov 03 08:15:05.221915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.221496 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp" event={"ID":"158efa57-2705-48b6-a778-f7afecc9fde5","Type":"ContainerDied","Data":"c37a8e4f3b8426f7a966f8406b12a64489689234de036dec0fea011bdd7a21a4"} Nov 03 08:15:05.221915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.221532 5951 scope.go:122] "RemoveContainer" containerID="8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407" Nov 03 08:15:05.224169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.224143 5951 generic.go:356] "Generic (PLEG): container finished" podID="60aafb5d-1519-4aa7-8135-01905662958b" containerID="9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061" exitCode=60 Nov 03 08:15:05.224402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.224324 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerDied","Data":"9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061"} Nov 03 08:15:05.224631 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.224613 5951 scope.go:122] "RemoveContainer" containerID="9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061" Nov 03 08:15:05.225102 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:05.225079 5951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf_odh-ai-gateway-infra(60aafb5d-1519-4aa7-8135-01905662958b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" podUID="60aafb5d-1519-4aa7-8135-01905662958b" Nov 03 08:15:05.231256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.231230 5951 generic.go:356] "Generic (PLEG): container finished" podID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerID="74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3" exitCode=60 Nov 03 08:15:05.231356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.231312 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerDied","Data":"74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3"} Nov 03 08:15:05.231545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.231525 5951 scope.go:122] "RemoveContainer" containerID="74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3" Nov 03 08:15:05.231849 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:05.231825 5951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx_odh-ai-gateway-infra(9a0effc8-50e2-4ea6-86be-0649ad5aab23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" Nov 03 08:15:05.237519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.236055 5951 scope.go:122] "RemoveContainer" containerID="8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407" Nov 03 08:15:05.237519 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:05.236371 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407\": container with ID starting with 8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407 not found: ID does not exist" containerID="8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407" Nov 03 08:15:05.237519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.236404 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407"} err="failed to get container status \"8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407\": rpc error: code = NotFound desc = could not find container \"8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407\": container with ID starting with 8fc372b8379ef5ef065517d95ac3c0c76a06844d478497e2ed2ce7b15b0ff407 not found: ID does not exist" Nov 03 08:15:05.237519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.236426 5951 scope.go:122] "RemoveContainer" containerID="69391bd1824a24ff82fe05c3363c707ee7005d24b918df35852f26bbd68705d3" Nov 03 08:15:05.258108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.254076 5951 scope.go:122] "RemoveContainer" containerID="a2a596a18fbdb9d6f129e77a8fc70305c84132ba037db83b95e56a759ac9d5c6" Nov 03 08:15:05.269142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.269111 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp"] Nov 03 08:15:05.274157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:05.274135 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94e3cb-786c85ffcf-n46hp"] Nov 03 08:15:06.317005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.316966 5951 scope.go:122] "RemoveContainer" containerID="9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061" Nov 03 08:15:06.317993 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:06.317963 5951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf_odh-ai-gateway-infra(60aafb5d-1519-4aa7-8135-01905662958b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" podUID="60aafb5d-1519-4aa7-8135-01905662958b" Nov 03 08:15:06.337263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.337208 5951 scope.go:122] "RemoveContainer" containerID="74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3" Nov 03 08:15:06.338601 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:06.338570 5951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx_odh-ai-gateway-infra(9a0effc8-50e2-4ea6-86be-0649ad5aab23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" Nov 03 08:15:06.581844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.581773 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj"] Nov 03 08:15:06.582703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.582686 5951 cpu_manager.go:414] "containerMap: removing container" podUID="158efa57-2705-48b6-a778-f7afecc9fde5" containerName="payload-processing" Nov 03 08:15:06.582703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.582702 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158efa57-2705-48b6-a778-f7afecc9fde5" containerName="payload-processing" Nov 03 08:15:06.582864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.582851 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="158efa57-2705-48b6-a778-f7afecc9fde5" containerName="payload-processing" Nov 03 08:15:06.588292 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.588270 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.599955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.599679 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj"] Nov 03 08:15:06.622056 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.621477 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9b559e-serving-cert\"" Nov 03 08:15:06.636259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.635986 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.636259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.636091 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t756f\" (UniqueName: \"kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.636259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.636138 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.636259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.636220 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.759177 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.759471 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:06.759488 5951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-serving-cert: secret "maas-api-e2e-inf-a-9b559e-serving-cert" not found Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:06.759577 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls podName:f15ad4bc-6867-4027-b055-bf9b62894b9a nodeName:}" failed. No retries permitted until 2026-11-03 08:15:07.259556641 +0000 UTC m=+1246.641007553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls") pod "maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" (UID: "f15ad4bc-6867-4027-b055-bf9b62894b9a") : secret "maas-api-e2e-inf-a-9b559e-serving-cert" not found Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.759728 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t756f\" (UniqueName: \"kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.759804 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.760678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.760352 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.771863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.771839 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t756f\" (UniqueName: \"kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.772167 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.772147 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:06.773140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:06.773116 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158efa57-2705-48b6-a778-f7afecc9fde5" path="/var/lib/kubelet/pods/158efa57-2705-48b6-a778-f7afecc9fde5/volumes" Nov 03 08:15:07.290407 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.290362 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:07.306123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.306093 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:07.522244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.522213 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:07.708201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.708169 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v"] Nov 03 08:15:07.724710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.720117 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:07.724710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.722265 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v"] Nov 03 08:15:07.780674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.779928 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-179c16-dockercfg-mslsr\"" Nov 03 08:15:07.780674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.780024 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-179c16\"" Nov 03 08:15:07.876927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.860428 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz\") pod \"payload-processing-e2e-inf-b-179c16-c957df445-9tz5v\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:07.876927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:07.860769 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-179c16-c957df445-9tz5v\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:08.079909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.079855 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj"] Nov 03 08:15:08.135965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.132540 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz\") pod \"payload-processing-e2e-inf-b-179c16-c957df445-9tz5v\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:08.135965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.133672 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-179c16-c957df445-9tz5v\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:08.135965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.134619 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-179c16-c957df445-9tz5v\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:08.149579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.149514 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz\") pod \"payload-processing-e2e-inf-b-179c16-c957df445-9tz5v\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:08.356418 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.356385 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:08.466717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.464454 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" event={"ID":"f15ad4bc-6867-4027-b055-bf9b62894b9a","Type":"ContainerStarted","Data":"236559920abfcc8d63df5e808fbf99f327c01a382c4a01e6750fc929cfea8a2c"} Nov 03 08:15:08.793339 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:15:08.793300 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd8f6e3_3644_469e_b73b_db7758756182.slice/crio-794d8a00033c1b503d213cc1afcbccecce7a2b80de0fa1df3d94daefc05198bb WatchSource:0}: Error finding container 794d8a00033c1b503d213cc1afcbccecce7a2b80de0fa1df3d94daefc05198bb: Status 404 returned error can't find the container with id 794d8a00033c1b503d213cc1afcbccecce7a2b80de0fa1df3d94daefc05198bb Nov 03 08:15:08.811178 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:08.811137 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v"] Nov 03 08:15:09.488283 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.488226 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" event={"ID":"f15ad4bc-6867-4027-b055-bf9b62894b9a","Type":"ContainerStarted","Data":"9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e"} Nov 03 08:15:09.488483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.488320 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:09.490794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.490747 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" event={"ID":"cdd8f6e3-3644-469e-b73b-db7758756182","Type":"ContainerStarted","Data":"6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a"} Nov 03 08:15:09.490794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.490797 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" event={"ID":"cdd8f6e3-3644-469e-b73b-db7758756182","Type":"ContainerStarted","Data":"794d8a00033c1b503d213cc1afcbccecce7a2b80de0fa1df3d94daefc05198bb"} Nov 03 08:15:09.491524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.491503 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:09.514681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.512792 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" podStartSLOduration=2.925527709 podStartE2EDuration="3.512769726s" totalImagesPullingTime="587.242017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:09.507210413 +0000 UTC m=+1248.888661331" watchObservedRunningTime="2026-11-03 08:15:09.512769726 +0000 UTC m=+1248.894220644" Nov 03 08:15:09.554261 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:09.554198 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" podStartSLOduration=2.554175569 podStartE2EDuration="2.554175569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:09.537811706 +0000 UTC m=+1248.919262626" watchObservedRunningTime="2026-11-03 08:15:09.554175569 +0000 UTC m=+1248.935626484" Nov 03 08:15:14.236881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:14.236846 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:15:15.540558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:15.540519 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:15:15.729796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:15.729758 5951 scope.go:122] "RemoveContainer" containerID="9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061" Nov 03 08:15:16.621345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:16.621090 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerStarted","Data":"99eef79537a4fdc96a9a38588d67152208a5844fd5dff4c0079e640c172063a5"} Nov 03 08:15:16.845839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:16.845807 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj"] Nov 03 08:15:16.846960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:16.846733 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" containerID="cri-o://f2dff48b5cf9caa812458e54c022d3272f50c200fd70a4731eb0f44311ba833a" gracePeriod=30 Nov 03 08:15:16.961355 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:16.961263 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4"] Nov 03 08:15:16.961610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:16.961565 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" containerID="cri-o://e314caae57f9d6686808b447139ee2c6677b86bc77c2d05333a82e95389a0b9e" gracePeriod=30 Nov 03 08:15:17.670703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:17.644186 5951 generic.go:356] "Generic (PLEG): container finished" podID="60aafb5d-1519-4aa7-8135-01905662958b" containerID="99eef79537a4fdc96a9a38588d67152208a5844fd5dff4c0079e640c172063a5" exitCode=60 Nov 03 08:15:17.670703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:17.644240 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerDied","Data":"99eef79537a4fdc96a9a38588d67152208a5844fd5dff4c0079e640c172063a5"} Nov 03 08:15:17.670703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:17.644300 5951 scope.go:122] "RemoveContainer" containerID="9c152eead0cff115152033af5dac0d8c48a1fa13f78d6e5ae686f82bc709f061" Nov 03 08:15:17.670703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:17.644895 5951 scope.go:122] "RemoveContainer" containerID="99eef79537a4fdc96a9a38588d67152208a5844fd5dff4c0079e640c172063a5" Nov 03 08:15:17.670703 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:17.645370 5951 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf_odh-ai-gateway-infra(60aafb5d-1519-4aa7-8135-01905662958b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" podUID="60aafb5d-1519-4aa7-8135-01905662958b" Nov 03 08:15:17.670703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:17.664637 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf"] Nov 03 08:15:17.790220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:17.787148 5951 scope.go:122] "RemoveContainer" containerID="74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3" Nov 03 08:15:18.660588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:18.660549 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerStarted","Data":"ad66ca0068a2a7b0db90d2b891785b8c303affe635d0a9c62017ed371fdeceef"} Nov 03 08:15:18.941139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:18.941110 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:19.071868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.071827 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:15:19.072173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.071890 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:15:19.085091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.085059 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca\") pod \"60aafb5d-1519-4aa7-8135-01905662958b\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " Nov 03 08:15:19.085369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.085349 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr\" (UniqueName: \"kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr\") pod \"60aafb5d-1519-4aa7-8135-01905662958b\" (UID: \"60aafb5d-1519-4aa7-8135-01905662958b\") " Nov 03 08:15:19.085497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.085471 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca" pod "60aafb5d-1519-4aa7-8135-01905662958b" (UID: "60aafb5d-1519-4aa7-8135-01905662958b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:19.085898 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.085880 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aafb5d-1519-4aa7-8135-01905662958b-maas-api-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:19.087762 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.087734 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr" pod "60aafb5d-1519-4aa7-8135-01905662958b" (UID: "60aafb5d-1519-4aa7-8135-01905662958b"). InnerVolumeSpecName "kube-api-access-phpwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:19.187944 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.187905 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phpwr\" (UniqueName: \"kubernetes.io/projected/60aafb5d-1519-4aa7-8135-01905662958b-kube-api-access-phpwr\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:19.697159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.697120 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx"] Nov 03 08:15:19.701230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.701197 5951 generic.go:356] "Generic (PLEG): container finished" podID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerID="ad66ca0068a2a7b0db90d2b891785b8c303affe635d0a9c62017ed371fdeceef" exitCode=60 Nov 03 08:15:19.701687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.701635 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerDied","Data":"ad66ca0068a2a7b0db90d2b891785b8c303affe635d0a9c62017ed371fdeceef"} Nov 03 08:15:19.701850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.701725 5951 scope.go:122] "RemoveContainer" containerID="74da09cd2b0564388f5b2633cdc9b3c41b3438b39dc6b4849843b02caadb57d3" Nov 03 08:15:19.711786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.709143 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" event={"ID":"60aafb5d-1519-4aa7-8135-01905662958b","Type":"ContainerDied","Data":"fd5fc135d506333c2e762cd62ed3759fa15e9c67c54741ce4931fada68b393d4"} Nov 03 08:15:19.711786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.709252 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf" Nov 03 08:15:19.716177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.714500 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k"] Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.716772 5951 cpu_manager.go:414] "containerMap: removing container" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.716794 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.716809 5951 cpu_manager.go:414] "containerMap: removing container" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.716833 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.716922 5951 cpu_manager.go:414] "containerMap: removing container" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.716931 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.717169 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.722134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.717182 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:19.725851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.725688 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.734886 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.734454 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e0f576-serving-cert\"" Nov 03 08:15:19.745023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.744986 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k"] Nov 03 08:15:19.747411 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.747384 5951 scope.go:122] "RemoveContainer" containerID="99eef79537a4fdc96a9a38588d67152208a5844fd5dff4c0079e640c172063a5" Nov 03 08:15:19.813324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.801725 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.813324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.801771 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.813324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.801802 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfj2\" (UniqueName: \"kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.813324 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.802011 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.815011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.814966 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf"] Nov 03 08:15:19.821042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.821013 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-bd085d-29894895-2cwnf"] Nov 03 08:15:19.912728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.912150 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.913248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.913217 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.913452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.913431 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.913575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.913558 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfj2\" (UniqueName: \"kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.915075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.915037 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.917613 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:19.915386 5951 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-serving-cert: secret "maas-api-e2e-inf-a-e0f576-serving-cert" not found Nov 03 08:15:19.917834 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:19.917820 5951 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls podName:388d8926-1579-4394-aa5f-fed4bee26395 nodeName:}" failed. No retries permitted until 2026-11-03 08:15:20.417793852 +0000 UTC m=+1259.799244767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls") pod "maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" (UID: "388d8926-1579-4394-aa5f-fed4bee26395") : secret "maas-api-e2e-inf-a-e0f576-serving-cert" not found Nov 03 08:15:19.954110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.954006 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:19.962507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:19.962425 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfj2\" (UniqueName: \"kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:20.056146 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.056112 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:20.222491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.220882 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct\" (UniqueName: \"kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct\") pod \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " Nov 03 08:15:20.222491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.221194 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca\") pod \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\" (UID: \"9a0effc8-50e2-4ea6-86be-0649ad5aab23\") " Nov 03 08:15:20.222491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.221517 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca" pod "9a0effc8-50e2-4ea6-86be-0649ad5aab23" (UID: "9a0effc8-50e2-4ea6-86be-0649ad5aab23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:20.222491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.222248 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0effc8-50e2-4ea6-86be-0649ad5aab23-maas-api-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:20.223779 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.223755 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct" pod "9a0effc8-50e2-4ea6-86be-0649ad5aab23" (UID: "9a0effc8-50e2-4ea6-86be-0649ad5aab23"). InnerVolumeSpecName "kube-api-access-257ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:20.323681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.323595 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257ct\" (UniqueName: \"kubernetes.io/projected/9a0effc8-50e2-4ea6-86be-0649ad5aab23-kube-api-access-257ct\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:20.453719 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.446033 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:20.495809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.487147 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\") pod \"maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:20.680502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.680467 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:20.791155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.791127 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" Nov 03 08:15:20.798508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.798474 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60aafb5d-1519-4aa7-8135-01905662958b" path="/var/lib/kubelet/pods/60aafb5d-1519-4aa7-8135-01905662958b/volumes" Nov 03 08:15:20.800329 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.800294 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx" event={"ID":"9a0effc8-50e2-4ea6-86be-0649ad5aab23","Type":"ContainerDied","Data":"2c05181cff5bc5bcdd2558b4c79593cc04c9583597cf424efea9c446286af3af"} Nov 03 08:15:20.800541 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.800525 5951 scope.go:122] "RemoveContainer" containerID="ad66ca0068a2a7b0db90d2b891785b8c303affe635d0a9c62017ed371fdeceef" Nov 03 08:15:20.821729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.818180 5951 generic.go:356] "Generic (PLEG): container finished" podID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerID="5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62" exitCode=0 Nov 03 08:15:20.821729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.818317 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" event={"ID":"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5","Type":"ContainerDied","Data":"5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62"} Nov 03 08:15:20.821729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.818905 5951 scope.go:122] "RemoveContainer" containerID="5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62" Nov 03 08:15:20.826316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.824496 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:15:20.826316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.824596 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:15:20.836803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.835660 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:15:20.954187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.954146 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx"] Nov 03 08:15:20.966001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:20.965977 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ba69a0-29894895-xcgvx"] Nov 03 08:15:21.117820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.112966 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k"] Nov 03 08:15:21.547530 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.547495 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:15:21.837615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.837527 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" event={"ID":"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5","Type":"ContainerStarted","Data":"7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10"} Nov 03 08:15:21.839738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.839704 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" event={"ID":"388d8926-1579-4394-aa5f-fed4bee26395","Type":"ContainerStarted","Data":"92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d"} Nov 03 08:15:21.839868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.839757 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" event={"ID":"388d8926-1579-4394-aa5f-fed4bee26395","Type":"ContainerStarted","Data":"9e9a50b91034d21bdb940622be303edd6c99b8519daf9950a3c240cdabe6ac07"} Nov 03 08:15:21.840597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.840577 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:21.879488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:21.879439 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" podStartSLOduration=2.286841344 podStartE2EDuration="2.879425546s" totalImagesPullingTime="592.584202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:21.874582507 +0000 UTC m=+1261.256033425" watchObservedRunningTime="2026-11-03 08:15:21.879425546 +0000 UTC m=+1261.260876463" Nov 03 08:15:22.767722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.755776 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 08:15:22.767722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.755837 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 08:15:22.962575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.962539 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" path="/var/lib/kubelet/pods/9a0effc8-50e2-4ea6-86be-0649ad5aab23/volumes" Nov 03 08:15:22.966918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.966875 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:15:22.967246 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.967203 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" event={"ID":"ba08e75a-59aa-49ae-97ed-ca66fe8387c6","Type":"ContainerDied","Data":"e314caae57f9d6686808b447139ee2c6677b86bc77c2d05333a82e95389a0b9e"} Nov 03 08:15:22.967447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.965849 5951 generic.go:356] "Generic (PLEG): container finished" podID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerID="e314caae57f9d6686808b447139ee2c6677b86bc77c2d05333a82e95389a0b9e" exitCode=0 Nov 03 08:15:22.971202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.971155 5951 generic.go:356] "Generic (PLEG): container finished" podID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerID="f2dff48b5cf9caa812458e54c022d3272f50c200fd70a4731eb0f44311ba833a" exitCode=0 Nov 03 08:15:22.972755 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:22.972730 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" event={"ID":"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef","Type":"ContainerDied","Data":"f2dff48b5cf9caa812458e54c022d3272f50c200fd70a4731eb0f44311ba833a"} Nov 03 08:15:23.149829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.149797 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:15:23.178749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.178706 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.178945 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.178863 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.179072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.178959 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.179072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.179003 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.179072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.179039 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.179949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.179881 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.181178 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.181013 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.181178 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.181124 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.181322 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.181292 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.181569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.181518 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy\") pod \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\" (UID: \"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef\") " Nov 03 08:15:23.181890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.181828 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:23.182346 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.182294 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.183040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.182947 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.183040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.182974 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.183040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.182988 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.183380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.183365 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.183895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.183856 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.186168 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.186125 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.186873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.186846 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "kube-api-access-twdgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:23.188334 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.188299 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:23.188497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.188477 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo" pod "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" (UID: "6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:15:23.277329 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.277296 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:15:23.289765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.287827 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.289765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.287864 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.289765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.287879 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.289765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.287893 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.289765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.287908 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdgv\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-kube-api-access-twdgv\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.289765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.287921 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.391413 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391327 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.391567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391478 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.391567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391520 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.391668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391590 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.391727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391710 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.391883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391866 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.391949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391921 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.392025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391952 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.392025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.391976 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.392133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.392038 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data\") pod \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\" (UID: \"ba08e75a-59aa-49ae-97ed-ca66fe8387c6\") " Nov 03 08:15:23.392278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.392256 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:23.392534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.392479 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.392737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.392712 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.393028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.393005 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.393099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.393033 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.393099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.393049 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.393099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.393063 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.393546 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.393519 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.453683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.451883 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:15:23.453683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.451888 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:23.454893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.454591 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:23.456768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.456742 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr" pod "ba08e75a-59aa-49ae-97ed-ca66fe8387c6" (UID: "ba08e75a-59aa-49ae-97ed-ca66fe8387c6"). InnerVolumeSpecName "kube-api-access-q7jnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:23.539235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.528395 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.539235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.528438 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.539235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.528454 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7jnr\" (UniqueName: \"kubernetes.io/projected/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-kube-api-access-q7jnr\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.539235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.528470 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.539235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.528486 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba08e75a-59aa-49ae-97ed-ca66fe8387c6-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:23.539235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.534719 5951 scope.go:122] "RemoveContainer" containerID="e314caae57f9d6686808b447139ee2c6677b86bc77c2d05333a82e95389a0b9e" Nov 03 08:15:23.552130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.552106 5951 scope.go:122] "RemoveContainer" containerID="f2dff48b5cf9caa812458e54c022d3272f50c200fd70a4731eb0f44311ba833a" Nov 03 08:15:23.741205 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.741171 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 08:15:23.741361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:23.741259 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 08:15:24.157819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.157784 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" Nov 03 08:15:24.168406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.159356 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4" event={"ID":"ba08e75a-59aa-49ae-97ed-ca66fe8387c6","Type":"ContainerDied","Data":"e00ffb918e3e07180aa44be55d634190fa6b169de6ced85175cf8366d416f761"} Nov 03 08:15:24.168826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.168796 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" event={"ID":"6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef","Type":"ContainerDied","Data":"edcfb11468cbba84e92f63ce0292b54a689c9d22b6c916da757c8a446d691ea1"} Nov 03 08:15:24.169018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.159403 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj" Nov 03 08:15:24.282486 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.282455 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4"] Nov 03 08:15:24.287464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.287435 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-nnzw4"] Nov 03 08:15:24.317732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.317495 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj"] Nov 03 08:15:24.374678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.374536 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-snshj"] Nov 03 08:15:24.744254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.744142 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 08:15:24.744254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.744217 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 08:15:24.746959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.746932 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" path="/var/lib/kubelet/pods/6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef/volumes" Nov 03 08:15:24.747540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:24.747520 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" path="/var/lib/kubelet/pods/ba08e75a-59aa-49ae-97ed-ca66fe8387c6/volumes" Nov 03 08:15:25.740159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:25.740126 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 08:15:25.740686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:25.740179 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 08:15:26.739840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:26.739808 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 08:15:26.740035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:26.739872 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 08:15:27.739854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:27.739822 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 08:15:27.740267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:27.739881 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 08:15:28.859833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:28.859697 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:15:28.861746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:28.861720 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:15:28.862861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:28.862844 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:15:29.165573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:29.165490 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:15:32.021992 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:32.021881 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:15:36.091946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.091908 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz"] Nov 03 08:15:36.092348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.092197 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" podUID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" containerName="maas-api" containerID="cri-o://e91dba1a1172d4222710a63eb28405599780a26264e5e9954adb6c2a27a6f245" gracePeriod=30 Nov 03 08:15:36.556670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.556614 5951 generic.go:356] "Generic (PLEG): container finished" podID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" containerID="e91dba1a1172d4222710a63eb28405599780a26264e5e9954adb6c2a27a6f245" exitCode=0 Nov 03 08:15:36.557237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.556952 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" event={"ID":"d06b5868-573b-4972-b1fd-9c6f681bf3aa","Type":"ContainerDied","Data":"e91dba1a1172d4222710a63eb28405599780a26264e5e9954adb6c2a27a6f245"} Nov 03 08:15:36.661024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.660994 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:15:36.762837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.762800 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls\") pod \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " Nov 03 08:15:36.763050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.763029 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7\" (UniqueName: \"kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7\") pod \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " Nov 03 08:15:36.763199 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.763183 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs\") pod \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " Nov 03 08:15:36.763302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.763281 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca\") pod \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\" (UID: \"d06b5868-573b-4972-b1fd-9c6f681bf3aa\") " Nov 03 08:15:36.764473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.763745 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca" pod "d06b5868-573b-4972-b1fd-9c6f681bf3aa" (UID: "d06b5868-573b-4972-b1fd-9c6f681bf3aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:36.764473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.764152 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06b5868-573b-4972-b1fd-9c6f681bf3aa-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:36.766070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.766037 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7" pod "d06b5868-573b-4972-b1fd-9c6f681bf3aa" (UID: "d06b5868-573b-4972-b1fd-9c6f681bf3aa"). InnerVolumeSpecName "kube-api-access-vxqt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:36.766240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.766221 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs" pod "d06b5868-573b-4972-b1fd-9c6f681bf3aa" (UID: "d06b5868-573b-4972-b1fd-9c6f681bf3aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:15:36.766690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.766619 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls" pod "d06b5868-573b-4972-b1fd-9c6f681bf3aa" (UID: "d06b5868-573b-4972-b1fd-9c6f681bf3aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:15:36.866424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.866349 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:36.866424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.866375 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d06b5868-573b-4972-b1fd-9c6f681bf3aa-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:36.866424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:36.866385 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqt7\" (UniqueName: \"kubernetes.io/projected/d06b5868-573b-4972-b1fd-9c6f681bf3aa-kube-api-access-vxqt7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:37.569170 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:37.569135 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" Nov 03 08:15:37.569170 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:37.569164 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" event={"ID":"d06b5868-573b-4972-b1fd-9c6f681bf3aa","Type":"ContainerDied","Data":"a4add3d8686a128db1ccce92e82f3640b07993cbed190c2b8e3f689a692408d3"} Nov 03 08:15:37.569808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:37.569230 5951 scope.go:122] "RemoveContainer" containerID="e91dba1a1172d4222710a63eb28405599780a26264e5e9954adb6c2a27a6f245" Nov 03 08:15:37.592361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:37.592327 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz"] Nov 03 08:15:37.599775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:37.599749 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz"] Nov 03 08:15:38.759403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:38.759363 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" path="/var/lib/kubelet/pods/d06b5868-573b-4972-b1fd-9c6f681bf3aa/volumes" Nov 03 08:15:39.447011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:39.446966 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94e3cb-59969d4c8f-227sz" podUID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.120:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:15:40.465885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.465851 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr"] Nov 03 08:15:40.466690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466670 5951 cpu_manager.go:414] "containerMap: removing container" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.466690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466686 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466723 5951 cpu_manager.go:414] "containerMap: removing container" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466728 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466739 5951 cpu_manager.go:414] "containerMap: removing container" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466744 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466756 5951 cpu_manager.go:414] "containerMap: removing container" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466760 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466770 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466775 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466787 5951 cpu_manager.go:414] "containerMap: removing container" podUID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" containerName="maas-api" Nov 03 08:15:40.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466791 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" containerName="maas-api" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466944 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466959 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466966 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aafb5d-1519-4aa7-8135-01905662958b" containerName="cleanup" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466974 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff0891c-7d56-4964-b6c0-1e92bd2ac1ef" containerName="istio-proxy" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466982 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0effc8-50e2-4ea6-86be-0649ad5aab23" containerName="cleanup" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466990 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba08e75a-59aa-49ae-97ed-ca66fe8387c6" containerName="istio-proxy" Nov 03 08:15:40.467130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.466996 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b5868-573b-4972-b1fd-9c6f681bf3aa" containerName="maas-api" Nov 03 08:15:40.472525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.472479 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.479094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.479067 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b8a152-dockercfg-qdkpw\"" Nov 03 08:15:40.479947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.479898 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b8a152\"" Nov 03 08:15:40.485176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.482976 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr"] Nov 03 08:15:40.530927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.530898 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892"] Nov 03 08:15:40.537579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.537291 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.537749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.537694 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qb6\" (UniqueName: \"kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6\") pod \"payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.538861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.538837 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.540235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.540206 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892"] Nov 03 08:15:40.662667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.651006 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw964\" (UniqueName: \"kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964\") pod \"payload-processing-e2e-inf-b-b8a152-76d844d85-rp892\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.662667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.651104 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qb6\" (UniqueName: \"kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6\") pod \"payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.662667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.651231 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.662667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.651332 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b8a152-76d844d85-rp892\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.662667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.661445 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.766717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.764032 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b8a152-76d844d85-rp892\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.766717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.764278 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw964\" (UniqueName: \"kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964\") pod \"payload-processing-e2e-inf-b-b8a152-76d844d85-rp892\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.766717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.765546 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b8a152-76d844d85-rp892\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.800676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.799853 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw964\" (UniqueName: \"kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964\") pod \"payload-processing-e2e-inf-b-b8a152-76d844d85-rp892\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:40.800676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.800483 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qb6\" (UniqueName: \"kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6\") pod \"payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.836673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.835931 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:40.916159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:40.915527 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:41.210618 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:15:41.210584 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3eff667_e577_4878_bf46_4fc8a264aef0.slice/crio-27007472d32f3a224f69e071d0829b6e05fc1a7432741d0b6eb50fff9d747a4e WatchSource:0}: Error finding container 27007472d32f3a224f69e071d0829b6e05fc1a7432741d0b6eb50fff9d747a4e: Status 404 returned error can't find the container with id 27007472d32f3a224f69e071d0829b6e05fc1a7432741d0b6eb50fff9d747a4e Nov 03 08:15:41.210982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.210961 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr"] Nov 03 08:15:41.221406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.221380 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892"] Nov 03 08:15:41.770511 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.770471 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" event={"ID":"c3eff667-e577-4878-bf46-4fc8a264aef0","Type":"ContainerStarted","Data":"50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d"} Nov 03 08:15:41.771009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.770542 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:41.771009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.770561 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" event={"ID":"c3eff667-e577-4878-bf46-4fc8a264aef0","Type":"ContainerStarted","Data":"27007472d32f3a224f69e071d0829b6e05fc1a7432741d0b6eb50fff9d747a4e"} Nov 03 08:15:41.772475 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.772439 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" event={"ID":"710222a7-fae0-41ae-be1e-0312f29fd512","Type":"ContainerStarted","Data":"05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac"} Nov 03 08:15:41.772589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.772493 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:41.772589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.772516 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" event={"ID":"710222a7-fae0-41ae-be1e-0312f29fd512","Type":"ContainerStarted","Data":"dbba44adc1de97efacbce0a9994ab852b78b96f0f28d2b79dad356ec5926afed"} Nov 03 08:15:41.788385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.788337 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" podStartSLOduration=1.7883211289999998 podStartE2EDuration="1.788321129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:41.784601037 +0000 UTC m=+1281.166051955" watchObservedRunningTime="2026-11-03 08:15:41.788321129 +0000 UTC m=+1281.169772047" Nov 03 08:15:41.800912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:41.800852 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" podStartSLOduration=1.800833049 podStartE2EDuration="1.800833049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:41.797609123 +0000 UTC m=+1281.179060038" watchObservedRunningTime="2026-11-03 08:15:41.800833049 +0000 UTC m=+1281.182283968" Nov 03 08:15:46.977603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:46.977563 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz"] Nov 03 08:15:46.978100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:46.977886 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" containerID="cri-o://419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641" gracePeriod=30 Nov 03 08:15:51.352715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.352619 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq"] Nov 03 08:15:51.359192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.359166 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.362981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.362952 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-616e46-dockercfg-hhhkq\"" Nov 03 08:15:51.366659 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.366620 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq"] Nov 03 08:15:51.431852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.427478 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-616e46\"" Nov 03 08:15:51.460585 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.460544 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.460965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.460944 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbb4\" (UniqueName: \"kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4\") pod \"payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.598670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.593986 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.634773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.625972 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.634773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.626112 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbb4\" (UniqueName: \"kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4\") pod \"payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.670344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.670313 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbb4\" (UniqueName: \"kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4\") pod \"payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:51.686258 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:51.685951 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:52.098948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.074434 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq"] Nov 03 08:15:52.786359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.786333 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:15:52.786750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.786397 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:15:52.796170 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.796140 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:15:52.865760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.865727 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.865937 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.865821 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.865937 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.865869 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.865946 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866138 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866118 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866159 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866190 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866284 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866321 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg\") pod \"88272ac9-229c-4604-96bc-e02d42140726\" (UID: \"88272ac9-229c-4604-96bc-e02d42140726\") " Nov 03 08:15:52.866436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866356 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:52.866497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866436 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:52.866591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.866568 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:52.867084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.867064 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.867196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.867087 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.867196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.867096 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.867196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.867100 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:52.867616 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.867520 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:15:52.870129 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.869801 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:15:52.870454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.870435 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "kube-api-access-82fmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:52.870792 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.870765 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:15:52.871817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.871792 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token" pod "88272ac9-229c-4604-96bc-e02d42140726" (UID: "88272ac9-229c-4604-96bc-e02d42140726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:15:52.945155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.945112 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" event={"ID":"a9a65b48-df70-4abb-bf90-32d68094d841","Type":"ContainerStarted","Data":"024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c"} Nov 03 08:15:52.945365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.945169 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" event={"ID":"a9a65b48-df70-4abb-bf90-32d68094d841","Type":"ContainerStarted","Data":"4582db3709205f072eb54d26b5c0890173d723a5ec6eba06f5a4d8a6c181af43"} Nov 03 08:15:52.945365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.945345 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:15:52.946823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.946785 5951 generic.go:356] "Generic (PLEG): container finished" podID="88272ac9-229c-4604-96bc-e02d42140726" containerID="419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641" exitCode=0 Nov 03 08:15:52.946968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.946832 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" Nov 03 08:15:52.946968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.946879 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" event={"ID":"88272ac9-229c-4604-96bc-e02d42140726","Type":"ContainerDied","Data":"419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641"} Nov 03 08:15:52.946968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.946936 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz" event={"ID":"88272ac9-229c-4604-96bc-e02d42140726","Type":"ContainerDied","Data":"bf63cd46285e47dd29a14e8a326ee7898a4673d47f35cf448ef9c7d4aed41f53"} Nov 03 08:15:52.946968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.946964 5951 scope.go:122] "RemoveContainer" containerID="419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641" Nov 03 08:15:52.961585 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.961560 5951 scope.go:122] "RemoveContainer" containerID="419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641" Nov 03 08:15:52.962026 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:15:52.961998 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641\": container with ID starting with 419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641 not found: ID does not exist" containerID="419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641" Nov 03 08:15:52.962116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.962040 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641"} err="failed to get container status \"419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641\": rpc error: code = NotFound desc = could not find container \"419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641\": container with ID starting with 419c582b157ac3116d055ba134e1876155d741c11cb715088edc9c29810bf641 not found: ID does not exist" Nov 03 08:15:52.964366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.964312 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" podStartSLOduration=1.96429975 podStartE2EDuration="1.96429975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:52.963839278 +0000 UTC m=+1292.345290196" watchObservedRunningTime="2026-11-03 08:15:52.96429975 +0000 UTC m=+1292.345750666" Nov 03 08:15:52.972025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.971992 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82fmg\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-kube-api-access-82fmg\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.972025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.972019 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88272ac9-229c-4604-96bc-e02d42140726-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.972025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.972034 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.972350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.972048 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88272ac9-229c-4604-96bc-e02d42140726-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.972350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.972065 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88272ac9-229c-4604-96bc-e02d42140726-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:52.972350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:52.972084 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88272ac9-229c-4604-96bc-e02d42140726-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:15:53.032153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:53.021816 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz"] Nov 03 08:15:53.079598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:53.079560 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-fl4qz"] Nov 03 08:15:54.781633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:54.781594 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88272ac9-229c-4604-96bc-e02d42140726" path="/var/lib/kubelet/pods/88272ac9-229c-4604-96bc-e02d42140726/volumes" Nov 03 08:15:54.856944 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:54.856909 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl"] Nov 03 08:15:54.872337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:54.870952 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d"] Nov 03 08:15:58.205668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.187502 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn"] Nov 03 08:15:58.205668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.189010 5951 cpu_manager.go:414] "containerMap: removing container" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" Nov 03 08:15:58.205668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.189035 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" Nov 03 08:15:58.205668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.189276 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="88272ac9-229c-4604-96bc-e02d42140726" containerName="istio-proxy" Nov 03 08:15:58.287343 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.287296 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn"] Nov 03 08:15:58.288805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.288215 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.301629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.301600 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-27c674-serving-cert\"" Nov 03 08:15:58.366865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.366824 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.377828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.376101 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.467707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.467600 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.468068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.468033 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspkm\" (UniqueName: \"kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.569464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.569427 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.569691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.569489 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.569691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.569560 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspkm\" (UniqueName: \"kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.569691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.569666 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.570591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.570567 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.572689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.572379 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.572689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.572418 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.577140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.577119 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspkm\" (UniqueName: \"kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm\") pod \"maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:58.800714 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:58.797571 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:15:59.820286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:59.820249 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn"] Nov 03 08:15:59.907143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:59.888752 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" podUID="d9a064df-d861-461f-8412-b6c7d05d4aa8" containerName="payload-pre-processing" containerID="cri-o://f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f" gracePeriod=25 Nov 03 08:15:59.935331 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:15:59.933362 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6024e94c_640b_4ccf_9e39_dd9bd115ba32.slice/crio-21699ea3576808ac85f5ea6604030efc119adefb31b7e7f294497acce9a12cc7 WatchSource:0}: Error finding container 21699ea3576808ac85f5ea6604030efc119adefb31b7e7f294497acce9a12cc7: Status 404 returned error can't find the container with id 21699ea3576808ac85f5ea6604030efc119adefb31b7e7f294497acce9a12cc7 Nov 03 08:15:59.936794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:15:59.936472 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" podUID="8ec928ec-d13e-4510-a8f2-21776ab47979" containerName="payload-processing" containerID="cri-o://47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53" gracePeriod=25 Nov 03 08:16:00.457175 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.457129 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:16:00.571946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.571909 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume\") pod \"d9a064df-d861-461f-8412-b6c7d05d4aa8\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " Nov 03 08:16:00.572602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.572249 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw\" (UniqueName: \"kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw\") pod \"d9a064df-d861-461f-8412-b6c7d05d4aa8\" (UID: \"d9a064df-d861-461f-8412-b6c7d05d4aa8\") " Nov 03 08:16:00.572602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.572557 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume" pod "d9a064df-d861-461f-8412-b6c7d05d4aa8" (UID: "d9a064df-d861-461f-8412-b6c7d05d4aa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:16:00.573050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.573010 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a064df-d861-461f-8412-b6c7d05d4aa8-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:00.579141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.579101 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw" pod "d9a064df-d861-461f-8412-b6c7d05d4aa8" (UID: "d9a064df-d861-461f-8412-b6c7d05d4aa8"). InnerVolumeSpecName "kube-api-access-p9jrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:16:00.586700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.586380 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:16:00.595703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.595633 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" event={"ID":"6024e94c-640b-4ccf-9e39-dd9bd115ba32","Type":"ContainerStarted","Data":"21699ea3576808ac85f5ea6604030efc119adefb31b7e7f294497acce9a12cc7"} Nov 03 08:16:00.597503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.597469 5951 generic.go:356] "Generic (PLEG): container finished" podID="d9a064df-d861-461f-8412-b6c7d05d4aa8" containerID="f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f" exitCode=0 Nov 03 08:16:00.597814 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.597793 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" event={"ID":"d9a064df-d861-461f-8412-b6c7d05d4aa8","Type":"ContainerDied","Data":"f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f"} Nov 03 08:16:00.597883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.597843 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" event={"ID":"d9a064df-d861-461f-8412-b6c7d05d4aa8","Type":"ContainerDied","Data":"075b6e296126f8492d0cb121dadb1d24a8ee0832cc99fb46ead9bacce2165646"} Nov 03 08:16:00.597883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.597873 5951 scope.go:122] "RemoveContainer" containerID="f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f" Nov 03 08:16:00.598042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.598029 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d" Nov 03 08:16:00.613356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.612763 5951 generic.go:356] "Generic (PLEG): container finished" podID="8ec928ec-d13e-4510-a8f2-21776ab47979" containerID="47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53" exitCode=0 Nov 03 08:16:00.613356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.612853 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" event={"ID":"8ec928ec-d13e-4510-a8f2-21776ab47979","Type":"ContainerDied","Data":"47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53"} Nov 03 08:16:00.613356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.612898 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" event={"ID":"8ec928ec-d13e-4510-a8f2-21776ab47979","Type":"ContainerDied","Data":"a7f3fb40b4b2c44e4c3120e9e4b98367219418dd8b7a471f5eb840ac92928dcd"} Nov 03 08:16:00.613356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.612969 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl" Nov 03 08:16:00.636025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.635904 5951 scope.go:122] "RemoveContainer" containerID="f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f" Nov 03 08:16:00.636621 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:16:00.636477 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f\": container with ID starting with f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f not found: ID does not exist" containerID="f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f" Nov 03 08:16:00.636621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.636511 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f"} err="failed to get container status \"f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f\": rpc error: code = NotFound desc = could not find container \"f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f\": container with ID starting with f2a7b902d2b43fa3d7b85f247c16cadde95406bd46e9b45d708b745d3dd69f2f not found: ID does not exist" Nov 03 08:16:00.636621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.636537 5951 scope.go:122] "RemoveContainer" containerID="47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53" Nov 03 08:16:00.655063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.655042 5951 scope.go:122] "RemoveContainer" containerID="47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53" Nov 03 08:16:00.655542 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:16:00.655512 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53\": container with ID starting with 47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53 not found: ID does not exist" containerID="47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53" Nov 03 08:16:00.655623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.655559 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53"} err="failed to get container status \"47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53\": rpc error: code = NotFound desc = could not find container \"47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53\": container with ID starting with 47b41a1cfdc1ae4139b2a211acfc9ed2c444bd1c59e3756293afd285cba47e53 not found: ID does not exist" Nov 03 08:16:00.677266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.676391 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume\") pod \"8ec928ec-d13e-4510-a8f2-21776ab47979\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " Nov 03 08:16:00.677266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.676770 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume" pod "8ec928ec-d13e-4510-a8f2-21776ab47979" (UID: "8ec928ec-d13e-4510-a8f2-21776ab47979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:16:00.677266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.677229 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs\" (UniqueName: \"kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs\") pod \"8ec928ec-d13e-4510-a8f2-21776ab47979\" (UID: \"8ec928ec-d13e-4510-a8f2-21776ab47979\") " Nov 03 08:16:00.678153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.678124 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec928ec-d13e-4510-a8f2-21776ab47979-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:00.678153 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.678153 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jrw\" (UniqueName: \"kubernetes.io/projected/d9a064df-d861-461f-8412-b6c7d05d4aa8-kube-api-access-p9jrw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:00.680376 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.680323 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d"] Nov 03 08:16:00.689880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.688558 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs" pod "8ec928ec-d13e-4510-a8f2-21776ab47979" (UID: "8ec928ec-d13e-4510-a8f2-21776ab47979"). InnerVolumeSpecName "kube-api-access-clgbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:16:00.702021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.701990 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-228695-77878d9f79-v8g6d"] Nov 03 08:16:00.746633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.746587 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a064df-d861-461f-8412-b6c7d05d4aa8" path="/var/lib/kubelet/pods/d9a064df-d861-461f-8412-b6c7d05d4aa8/volumes" Nov 03 08:16:00.779826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.779795 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clgbs\" (UniqueName: \"kubernetes.io/projected/8ec928ec-d13e-4510-a8f2-21776ab47979-kube-api-access-clgbs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:00.940757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.940591 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl"] Nov 03 08:16:00.950122 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:00.950094 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-228695-596fcfc44f-ggdbl"] Nov 03 08:16:01.633272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:01.632320 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" event={"ID":"6024e94c-640b-4ccf-9e39-dd9bd115ba32","Type":"ContainerStarted","Data":"e24789bcb833eff4362724910b31fea73453d55bd2ec0602d19f48eeafe21103"} Nov 03 08:16:01.633272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:01.632468 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:16:01.654895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:01.654829 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" podStartSLOduration=3.12500111 podStartE2EDuration="3.654808887s" totalImagesPullingTime="529.807777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:01.647825818 +0000 UTC m=+1301.029276736" watchObservedRunningTime="2026-11-03 08:16:01.654808887 +0000 UTC m=+1301.036259805" Nov 03 08:16:01.963728 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:16:01.963683 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:16:02.753985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:02.753950 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec928ec-d13e-4510-a8f2-21776ab47979" path="/var/lib/kubelet/pods/8ec928ec-d13e-4510-a8f2-21776ab47979/volumes" Nov 03 08:16:04.098707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.098628 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.301269 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr"] Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.302411 5951 cpu_manager.go:414] "containerMap: removing container" podUID="8ec928ec-d13e-4510-a8f2-21776ab47979" containerName="payload-processing" Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.302431 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec928ec-d13e-4510-a8f2-21776ab47979" containerName="payload-processing" Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.302448 5951 cpu_manager.go:414] "containerMap: removing container" podUID="d9a064df-d861-461f-8412-b6c7d05d4aa8" containerName="payload-pre-processing" Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.302456 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a064df-d861-461f-8412-b6c7d05d4aa8" containerName="payload-pre-processing" Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.302589 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec928ec-d13e-4510-a8f2-21776ab47979" containerName="payload-processing" Nov 03 08:16:04.305746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.302598 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a064df-d861-461f-8412-b6c7d05d4aa8" containerName="payload-pre-processing" Nov 03 08:16:04.356277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.356182 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr"] Nov 03 08:16:04.356449 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.356307 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.359495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.359464 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b8a152-serving-cert\"" Nov 03 08:16:04.465149 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.465102 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.465323 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.465179 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsqbn\" (UniqueName: \"kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.465370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.465324 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.465433 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.465385 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.567479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.567208 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.567479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.567290 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsqbn\" (UniqueName: \"kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.567479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.567401 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.567479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.567468 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.569007 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.568974 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.570828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.570808 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.572393 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.572371 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.575906 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.575877 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsqbn\" (UniqueName: \"kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn\") pod \"maas-api-e2e-inf-b-b8a152-599648c95b-4skrr\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:04.695328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.695238 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:05.005516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:04.998661 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr"] Nov 03 08:16:05.253616 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.253520 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:16:05.254130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.253614 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:05.564242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.561819 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:16:05.564242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.561896 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:05.685552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.685516 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" event={"ID":"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6","Type":"ContainerStarted","Data":"2d199dd1e2776ee325e7c18a0d9d1e161dc0e3ea68ab8833ed07793978a1e22f"} Nov 03 08:16:05.687763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.687733 5951 generic.go:356] "Generic (PLEG): container finished" podID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerID="791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e" exitCode=0 Nov 03 08:16:05.687899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.687811 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" event={"ID":"a4853d6b-f6ba-4245-89aa-2069c46cea2a","Type":"ContainerDied","Data":"791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e"} Nov 03 08:16:05.688389 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.688370 5951 scope.go:122] "RemoveContainer" containerID="791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e" Nov 03 08:16:05.690614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.690581 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:05.690803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.690726 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:05.690803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:05.690779 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:06.746692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.746616 5951 generic.go:356] "Generic (PLEG): container finished" podID="8e71c051-5971-42b9-80a7-6b85964c8350" containerID="d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067" exitCode=0 Nov 03 08:16:06.748378 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.748358 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:06.748453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.748407 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" event={"ID":"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6","Type":"ContainerStarted","Data":"35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f"} Nov 03 08:16:06.748453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.748428 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" event={"ID":"a4853d6b-f6ba-4245-89aa-2069c46cea2a","Type":"ContainerStarted","Data":"51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667"} Nov 03 08:16:06.748453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.748446 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" event={"ID":"8e71c051-5971-42b9-80a7-6b85964c8350","Type":"ContainerDied","Data":"d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067"} Nov 03 08:16:06.748959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.748937 5951 scope.go:122] "RemoveContainer" containerID="d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067" Nov 03 08:16:06.751615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.751588 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:06.751869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.751849 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:06.752042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.752013 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:06.756043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:06.755977 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" podStartSLOduration=2.253003086 podStartE2EDuration="2.755964176s" totalImagesPullingTime="502.96109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:06.753795253 +0000 UTC m=+1306.135246194" watchObservedRunningTime="2026-11-03 08:16:06.755964176 +0000 UTC m=+1306.137415086" Nov 03 08:16:07.347683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.341047 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:16:07.347683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.342131 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 08:16:07.347683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.342185 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 08:16:07.654294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.654222 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:16:07.781995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.781965 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" event={"ID":"8e71c051-5971-42b9-80a7-6b85964c8350","Type":"ContainerStarted","Data":"cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036"} Nov 03 08:16:07.959788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.959708 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:16:07.959932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.959903 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:07.959975 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:07.959941 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:08.343729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:08.341989 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 08:16:08.343729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:08.342072 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 08:16:08.960749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:08.960701 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:08.961191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:08.960773 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:09.359809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:09.359762 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 08:16:09.360042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:09.359843 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 08:16:09.962677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:09.962097 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:09.962677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:09.962186 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:10.348676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.345101 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 08:16:10.348676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.345168 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 08:16:10.870960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.870920 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f"] Nov 03 08:16:10.878409 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.877393 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.883219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.883091 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e0f576-openshift-default-dockercfg-wnxjc\"" Nov 03 08:16:10.902265 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.893894 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f"] Nov 03 08:16:10.937514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.937469 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.938173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.938138 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.938381 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.938361 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.938685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.938663 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpxk\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.938864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.938847 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.939052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.939035 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.939200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.939183 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.939344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.939329 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.939463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.939447 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:10.966140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.966097 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:10.966757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:10.966725 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074386 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpxk\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074473 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074564 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074681 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074730 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074769 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.074957 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.075045 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.077671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.075097 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.079328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.079301 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.079990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.079959 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.080924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.080520 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.080924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.080590 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.080924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.080787 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.083751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.083451 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.084933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.084851 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.085728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.085706 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.087486 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.087463 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpxk\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.426831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.387618 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 08:16:11.426831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.387736 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 08:16:11.486673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.476009 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:11.977379 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.960128 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:11.977379 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:11.960190 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:12.384677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.382205 5951 patch_prober.go:29] interesting pod/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 08:16:12.384677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.382277 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 08:16:12.503691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.501713 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:12.503691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.501846 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:12.503691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.501906 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:12.650754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.650009 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f"] Nov 03 08:16:12.854239 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.853862 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:16:12.966721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.963806 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:12.966721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.963868 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:12.966721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.965553 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" event={"ID":"fd5e3cdd-09d6-40d8-b479-1179b58a9c27","Type":"ContainerStarted","Data":"af9535f52ad7b76d8f85f2564787fb14e5190a4aac60d359b0553d8540b57a5a"} Nov 03 08:16:13.004867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:12.999340 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podStartSLOduration=2.999316614 podStartE2EDuration="2.999316614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:12.995140504 +0000 UTC m=+1312.376591423" watchObservedRunningTime="2026-11-03 08:16:12.999316614 +0000 UTC m=+1312.380767524" Nov 03 08:16:13.346794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.346757 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:16:13.347524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.347498 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:16:13.348313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.348289 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:16:13.478023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.477998 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:13.479838 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.479811 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:13.479980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.479881 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:13.971678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.970344 5951 patch_prober.go:29] interesting pod/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 08:16:13.971678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.970418 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 08:16:13.991716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:13.991637 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" event={"ID":"fd5e3cdd-09d6-40d8-b479-1179b58a9c27","Type":"ContainerStarted","Data":"7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861"} Nov 03 08:16:14.341257 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.341220 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw"] Nov 03 08:16:14.346762 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.346735 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.354296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.354269 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw"] Nov 03 08:16:14.449807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.449776 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b559e-openshift-default-dockercfg-6hqmb\"" Nov 03 08:16:14.478668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.478613 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:14.478827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.478672 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:14.480553 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.480525 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.480711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.480582 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.480773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.480706 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.480773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.480760 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.480929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.480910 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.481041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.481027 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hz28\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.481154 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.481141 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.481219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.481188 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.481219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.481212 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583154 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583389 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583222 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583389 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583286 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583389 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583376 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hz28\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583462 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583492 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583564 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583508 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583599 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583638 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583731 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583914 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583882 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583935 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.583977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.583958 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.584421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.584398 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.586035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.586015 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.586218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.586201 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.590897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.590861 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.591024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.591008 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hz28\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.671839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.671751 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:14.969567 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.969494 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:16:14.970357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.970173 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:16:14.970789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:14.970769 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:16:15.056016 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.054347 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:15.056016 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.054429 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:15.056016 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.054472 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:15.056903 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.056865 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw"] Nov 03 08:16:15.159342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.159315 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl"] Nov 03 08:16:15.193465 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.193438 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl"] Nov 03 08:16:15.193696 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.193549 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.280007 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.279973 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-228695-openshift-default-dockercfg-9dgpq\"" Nov 03 08:16:15.338006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.337967 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.338285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.338260 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.338417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.338401 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.338608 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.338590 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnpz\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.338780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.338763 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.338968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.338949 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.339100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.339086 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.339342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.339325 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.339520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.339506 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442291 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442388 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442482 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442518 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442556 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442682 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnpz\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442733 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442803 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.442901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.442868 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.443306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.443107 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.443732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.443704 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.444133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.443985 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.444618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.444577 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.445268 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.445242 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.447228 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.447204 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.447480 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.447463 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.450476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.450445 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnpz\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.450828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.450807 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-prtbl\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.478899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.478824 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:15.479031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.478903 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:15.553850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.553818 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:15.935831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.934979 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl"] Nov 03 08:16:15.957563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.957499 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:15.957728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.957570 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:15.957728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:15.957600 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:16.044323 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.044281 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" event={"ID":"dd2df007-fdd1-4caa-b7e7-33477c96e02b","Type":"ContainerStarted","Data":"ff45e8208f0978021b9b2563eef9723934a2a35e727e1b9d3dec5c6c8d4b6be1"} Nov 03 08:16:16.044516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.044503 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" event={"ID":"dd2df007-fdd1-4caa-b7e7-33477c96e02b","Type":"ContainerStarted","Data":"c61e68f144c6cbfa431ad409699845df28566484b3f5653bea794654f96700a3"} Nov 03 08:16:16.048571 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.048549 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" event={"ID":"dc41e7dc-d51b-4bc4-a217-26026641f277","Type":"ContainerStarted","Data":"c761f12cb09afae6f6333d60884139e94e1a1faa737cf83d7481d5dc41d4e35c"} Nov 03 08:16:16.069907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.069847 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podStartSLOduration=2.069825196 podStartE2EDuration="2.069825196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:16.066425866 +0000 UTC m=+1315.447876796" watchObservedRunningTime="2026-11-03 08:16:16.069825196 +0000 UTC m=+1315.451276114" Nov 03 08:16:16.479829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.479784 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:16.480341 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.479844 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:16.674100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.674067 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:16.694560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.694523 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 08:16:16.704924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:16.704887 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 08:16:17.382766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.382727 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" event={"ID":"dc41e7dc-d51b-4bc4-a217-26026641f277","Type":"ContainerStarted","Data":"0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe"} Nov 03 08:16:17.431680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.409381 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podStartSLOduration=2.409364274 podStartE2EDuration="2.409364274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:17.407916905 +0000 UTC m=+1316.789367842" watchObservedRunningTime="2026-11-03 08:16:17.409364274 +0000 UTC m=+1316.790815192" Nov 03 08:16:17.498670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.492211 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:17.498670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.492281 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:17.597681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.596846 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:17.623793 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.623757 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:17.624051 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.624020 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:17.675216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.675133 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 08:16:17.675455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:17.675429 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 08:16:18.266784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.266734 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:18.406677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.406153 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:18.479669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.478805 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:18.479669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.478863 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:18.557080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.557040 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:18.561761 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.561707 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:18.659224 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.640080 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 08:16:18.659961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.659518 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:16:18.684674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.682265 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 08:16:18.684674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.682322 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 08:16:18.684674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.682758 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:18.772741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.771363 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 08:16:18.798670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:18.798071 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.248681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.240049 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.284302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.284257 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.284866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.284838 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:16:19.479691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.413170 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.479691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.413626 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:16:19.583670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.573688 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:19.583670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.573751 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:19.675038 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.669754 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.677924 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.677825 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:16:19.683883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.682576 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:16:19.689788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.689742 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.691015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.690992 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:16:19.711912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.711866 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 08:16:19.713233 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.713201 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 08:16:19.769661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.763075 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 08:16:19.769661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.764964 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 08:16:19.769661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.767175 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.818900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.818852 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.822287 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.822247 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:19.823558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.823537 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:16:19.823778 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:19.823766 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:16:20.255537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.255495 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:16:20.297670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.291417 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:16:20.376102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.373487 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 08:16:20.376102 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.374668 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:16:20.441673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.416257 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:16:20.485689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.485300 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:16:20.658671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.566118 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:20.658671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.566183 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:20.658671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.566840 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:20.658671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.568513 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:20.658671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.569113 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:16:20.658671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.644198 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:16:20.705367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.704898 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 08:16:20.705679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.705627 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 08:16:20.768261 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:20.756231 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:16:21.162727 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.162697 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:16:21.162960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.162948 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:16:21.163106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.163096 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:16:21.557672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.555930 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:21.557672 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.555976 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:21.672241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.672082 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 08:16:21.672241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:21.672141 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 08:16:22.555671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:22.554248 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:22.555671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:22.554325 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:22.678074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:22.678034 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:22.678992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:22.678961 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:22.679894 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:22.679872 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:16:23.554738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:23.554703 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-prtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 08:16:23.554933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:23.554759 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 08:16:24.562199 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:24.562165 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:24.563505 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:24.563478 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:24.564491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:24.564471 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:16:25.893683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:25.893037 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt"] Nov 03 08:16:25.924202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:25.921989 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:25.962669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:25.960534 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt"] Nov 03 08:16:26.064415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064320 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntffg\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.064630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064572 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.064740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064631 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.064792 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064741 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.064844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064798 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.064929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064911 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.064987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.064954 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.065115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.065098 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.065237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.065190 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.253821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.253729 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.253991 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.253849 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.253991 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.253901 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.254090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.253993 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.254090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.254027 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.254185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.254152 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.254312 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.254268 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.254440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.254360 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntffg\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.254570 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.254541 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.261065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.257313 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.261065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.257590 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.261065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.257930 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.261065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.258566 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.261065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.258833 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.264466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.264408 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.269002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.268935 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.269002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.268947 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntffg\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.269313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.269290 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.314520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.314487 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:26.679926 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:16:26.679891 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7615b3e_68c7_4e1d_9230_30be0de2d060.slice/crio-adbcdb38381f58e94c2e1ba06974fdb6922315b3ed7b8616f2cfed30e55b6a0e WatchSource:0}: Error finding container adbcdb38381f58e94c2e1ba06974fdb6922315b3ed7b8616f2cfed30e55b6a0e: Status 404 returned error can't find the container with id adbcdb38381f58e94c2e1ba06974fdb6922315b3ed7b8616f2cfed30e55b6a0e Nov 03 08:16:26.681360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.681322 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt"] Nov 03 08:16:26.683070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.683031 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:26.683187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.683113 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:26.683187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:26.683155 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:27.442846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:27.442808 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" event={"ID":"c7615b3e-68c7-4e1d-9230-30be0de2d060","Type":"ContainerStarted","Data":"3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9"} Nov 03 08:16:27.443361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:27.442876 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" event={"ID":"c7615b3e-68c7-4e1d-9230-30be0de2d060","Type":"ContainerStarted","Data":"adbcdb38381f58e94c2e1ba06974fdb6922315b3ed7b8616f2cfed30e55b6a0e"} Nov 03 08:16:27.463615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:27.463556 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podStartSLOduration=2.46354258 podStartE2EDuration="2.46354258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:27.459345272 +0000 UTC m=+1326.840796189" watchObservedRunningTime="2026-11-03 08:16:27.46354258 +0000 UTC m=+1326.844993497" Nov 03 08:16:28.319677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:28.319461 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:28.319677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:28.319605 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 08:16:28.319962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:28.319685 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 08:16:29.327798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.327095 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 08:16:29.327798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.327175 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 08:16:29.356786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.356750 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl"] Nov 03 08:16:29.362408 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.362384 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.374059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.374024 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl"] Nov 03 08:16:29.389430 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.389401 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84"] Nov 03 08:16:29.442177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.442141 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.456942 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.456912 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84"] Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459521 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll7r\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459610 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459635 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459716 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459739 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459828 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.459921 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.460058 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.460250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.460133 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.567113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567072 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.567284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567138 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.567284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567266 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.567451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567430 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.567541 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567524 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.567600 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567558 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.567680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567597 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.567680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567630 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.567800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567775 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll7r\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.567857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.567836 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.568223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.568200 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6rs\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.568912 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.568891 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.569013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.568557 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.569222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.569200 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.570380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.570013 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.570559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.570541 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.571171 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.570412 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.571344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.571326 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.571913 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.571895 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.572166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.572150 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.572803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.572777 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.573120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.573101 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.573600 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.573494 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.579768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.579516 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.579768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.579683 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll7r\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.580353 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.580338 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.584306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.583818 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.681902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.681864 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.681918 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.681941 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.682039 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6rs\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.682091 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.682179 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.682219 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.682245 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.682337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.682278 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.683123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.683088 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.683247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.683147 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.683247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.683162 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.683400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.683378 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.683587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.683567 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.684977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.684946 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.685292 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.685277 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.686573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.686551 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:29.690046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.690023 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.690142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.690073 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6rs\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:29.781425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:29.781095 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:30.162004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.161787 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl"] Nov 03 08:16:30.171336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.171277 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:30.171969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.171503 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:30.171969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.171543 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:30.321670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.316767 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 08:16:30.321670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.316830 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 08:16:30.383561 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.383066 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84"] Nov 03 08:16:30.411317 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:16:30.403474 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3b8e7c_fa60_4bbc_b919_5f7fcad02bf7.slice/crio-995a5da07c83e79a3c8adf8d5034c6b352b41797881d47ee0603c1bf0187dcc2 WatchSource:0}: Error finding container 995a5da07c83e79a3c8adf8d5034c6b352b41797881d47ee0603c1bf0187dcc2: Status 404 returned error can't find the container with id 995a5da07c83e79a3c8adf8d5034c6b352b41797881d47ee0603c1bf0187dcc2 Nov 03 08:16:30.467006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.450324 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:30.467006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.450412 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:30.467006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.450446 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:16:30.603671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.603588 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" event={"ID":"545ab5b3-977a-4507-909e-6fbaedc3877f","Type":"ContainerStarted","Data":"bc9698e21c4b95531d254f9d04c75fa609d59dfa93d1251bbe743b4a3ce9b0cc"} Nov 03 08:16:30.607668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.604803 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" event={"ID":"545ab5b3-977a-4507-909e-6fbaedc3877f","Type":"ContainerStarted","Data":"fc55b35b43dafa1f43865b044139113e14492b39fdd3a12a9d60f090b90f9bab"} Nov 03 08:16:30.622089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.622050 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" event={"ID":"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7","Type":"ContainerStarted","Data":"995a5da07c83e79a3c8adf8d5034c6b352b41797881d47ee0603c1bf0187dcc2"} Nov 03 08:16:30.652425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.652359 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podStartSLOduration=1.652334293 podStartE2EDuration="1.652334293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:30.63340288 +0000 UTC m=+1330.014853798" watchObservedRunningTime="2026-11-03 08:16:30.652334293 +0000 UTC m=+1330.033785216" Nov 03 08:16:30.695844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:30.695752 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:31.325860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.317035 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 08:16:31.325860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.317099 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 08:16:31.639351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.639255 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" event={"ID":"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7","Type":"ContainerStarted","Data":"e8a1dc870c69bdcdbde7a5642d218c9c64ff9d23b9f909b22e8958fbe9409252"} Nov 03 08:16:31.661614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.661549 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podStartSLOduration=2.661531668 podStartE2EDuration="2.661531668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:31.658872942 +0000 UTC m=+1331.040323860" watchObservedRunningTime="2026-11-03 08:16:31.661531668 +0000 UTC m=+1331.042982584" Nov 03 08:16:31.688860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.688824 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:31.689054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.688883 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:31.781245 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.781197 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:31.783514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.783485 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:31.783664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:31.783546 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:31.961297 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:16:31.961205 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:16:32.327188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:32.319949 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 08:16:32.327188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:32.320014 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 08:16:32.687050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:32.686962 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:32.687050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:32.687030 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:32.782406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:32.782373 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:32.782554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:32.782440 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:33.317669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:33.315501 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 08:16:33.317669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:33.315577 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 08:16:33.687939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:33.687837 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:33.687939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:33.687915 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:33.781999 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:33.781962 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:33.782174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:33.782022 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:34.467499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.466207 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:34.482677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.468121 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:34.489963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.489933 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:16:34.687351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.687238 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:34.687977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.687310 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:34.781815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.781777 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:34.781993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:34.781871 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:35.695677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:35.687559 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:35.695677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:35.687625 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:35.783676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:35.782749 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:35.783676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:35.782813 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:36.692775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:36.690537 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:36.692775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:36.690599 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:36.782325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:36.782292 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:36.782734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:36.782343 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:37.687670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:37.687623 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:37.687900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:37.687876 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:37.781621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:37.781586 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:37.781821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:37.781660 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:38.687341 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:38.687308 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:38.687791 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:38.687360 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:38.781955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:38.781924 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:38.782148 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:38.781974 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:39.687610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:39.687580 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:39.688186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:39.688158 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:39.782416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:39.782384 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:39.782615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:39.782436 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:40.687469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:40.687434 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 08:16:40.687713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:40.687512 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 08:16:40.793623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:40.793580 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:40.799676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:40.799073 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:41.693182 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:41.693147 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:41.697669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:41.693675 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:41.706668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:41.702856 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:16:41.784750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:41.784398 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 08:16:41.784750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:41.784513 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 08:16:42.787794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:42.787765 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:42.788237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:42.788165 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:42.789288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:42.789268 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:16:47.953713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:47.953603 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8kjgc"] Nov 03 08:16:47.954254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:47.953868 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-8kjgc" podUID="297faaa7-4afc-4bfb-b62c-251b209be679" containerName="manager" containerID="cri-o://8ba73ca46313e24e19b59342b788a31c3d4995f571f59ade4729cd83553de0bb" gracePeriod=10 Nov 03 08:16:49.147037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.146734 5951 generic.go:356] "Generic (PLEG): container finished" podID="297faaa7-4afc-4bfb-b62c-251b209be679" containerID="8ba73ca46313e24e19b59342b788a31c3d4995f571f59ade4729cd83553de0bb" exitCode=0 Nov 03 08:16:49.147037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.146956 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8kjgc" event={"ID":"297faaa7-4afc-4bfb-b62c-251b209be679","Type":"ContainerDied","Data":"8ba73ca46313e24e19b59342b788a31c3d4995f571f59ade4729cd83553de0bb"} Nov 03 08:16:49.205501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.205317 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 08:16:49.282028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.281998 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw\" (UniqueName: \"kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw\") pod \"297faaa7-4afc-4bfb-b62c-251b209be679\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " Nov 03 08:16:49.282201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.282038 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert\") pod \"297faaa7-4afc-4bfb-b62c-251b209be679\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " Nov 03 08:16:49.282201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.282144 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\") pod \"297faaa7-4afc-4bfb-b62c-251b209be679\" (UID: \"297faaa7-4afc-4bfb-b62c-251b209be679\") " Nov 03 08:16:49.285110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.285076 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs" pod "297faaa7-4afc-4bfb-b62c-251b209be679" (UID: "297faaa7-4afc-4bfb-b62c-251b209be679"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:16:49.285244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.285111 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert" pod "297faaa7-4afc-4bfb-b62c-251b209be679" (UID: "297faaa7-4afc-4bfb-b62c-251b209be679"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:16:49.288976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.288951 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw" pod "297faaa7-4afc-4bfb-b62c-251b209be679" (UID: "297faaa7-4afc-4bfb-b62c-251b209be679"). InnerVolumeSpecName "kube-api-access-dhbnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:16:49.384895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.384805 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbnw\" (UniqueName: \"kubernetes.io/projected/297faaa7-4afc-4bfb-b62c-251b209be679-kube-api-access-dhbnw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:49.384895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.384847 5951 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:49.384895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:49.384862 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297faaa7-4afc-4bfb-b62c-251b209be679-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:16:50.158226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:50.158195 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8kjgc" event={"ID":"297faaa7-4afc-4bfb-b62c-251b209be679","Type":"ContainerDied","Data":"b70f52d2f026287c9810a395e928547729aff9fb41d70d54212adb5b4b62c99b"} Nov 03 08:16:50.158226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:50.158226 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8kjgc" Nov 03 08:16:50.158800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:50.158258 5951 scope.go:122] "RemoveContainer" containerID="8ba73ca46313e24e19b59342b788a31c3d4995f571f59ade4729cd83553de0bb" Nov 03 08:16:50.186626 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:50.186594 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8kjgc"] Nov 03 08:16:50.191456 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:50.191429 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8kjgc"] Nov 03 08:16:50.741548 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:50.741513 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297faaa7-4afc-4bfb-b62c-251b209be679" path="/var/lib/kubelet/pods/297faaa7-4afc-4bfb-b62c-251b209be679/volumes" Nov 03 08:16:57.647917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.647879 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-fzvxh"] Nov 03 08:16:57.649115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.649089 5951 cpu_manager.go:414] "containerMap: removing container" podUID="297faaa7-4afc-4bfb-b62c-251b209be679" containerName="manager" Nov 03 08:16:57.649115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.649115 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297faaa7-4afc-4bfb-b62c-251b209be679" containerName="manager" Nov 03 08:16:57.649374 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.649358 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="297faaa7-4afc-4bfb-b62c-251b209be679" containerName="manager" Nov 03 08:16:57.655054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.655028 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.660474 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.660448 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wdxdf\"" Nov 03 08:16:57.660741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.660721 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 08:16:57.660843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.660807 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 08:16:57.661958 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.661933 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fzvxh"] Nov 03 08:16:57.741557 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.741513 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-metrics-certs\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.741847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.741706 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-cert\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.741847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.741825 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7g6j\" (UniqueName: \"kubernetes.io/projected/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-kube-api-access-m7g6j\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.843547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.843487 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-metrics-certs\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.843772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.843663 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-cert\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.843845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.843778 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7g6j\" (UniqueName: \"kubernetes.io/projected/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-kube-api-access-m7g6j\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.846438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.846410 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-cert\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.846604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.846467 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-metrics-certs\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.850769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.850743 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7g6j\" (UniqueName: \"kubernetes.io/projected/adcc9576-5d32-40f5-8c27-aa7a320ee7f1-kube-api-access-m7g6j\") pod \"odh-model-controller-769875db56-fzvxh\" (UID: \"adcc9576-5d32-40f5-8c27-aa7a320ee7f1\") " pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:57.985145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:57.985044 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:58.340867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:58.340832 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fzvxh"] Nov 03 08:16:58.342213 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:16:58.342183 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcc9576_5d32_40f5_8c27_aa7a320ee7f1.slice/crio-4368ba19b9ae99e67beea6cea2047f0890b3ab1ca3b64dfcef209635c44b3444 WatchSource:0}: Error finding container 4368ba19b9ae99e67beea6cea2047f0890b3ab1ca3b64dfcef209635c44b3444: Status 404 returned error can't find the container with id 4368ba19b9ae99e67beea6cea2047f0890b3ab1ca3b64dfcef209635c44b3444 Nov 03 08:16:59.260120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:59.260009 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fzvxh" event={"ID":"adcc9576-5d32-40f5-8c27-aa7a320ee7f1","Type":"ContainerStarted","Data":"a454c064357a5bc081852f3400ee767d2b6e7fdb10367b16ffc4ebccb377d7f8"} Nov 03 08:16:59.260120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:59.260088 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fzvxh" event={"ID":"adcc9576-5d32-40f5-8c27-aa7a320ee7f1","Type":"ContainerStarted","Data":"4368ba19b9ae99e67beea6cea2047f0890b3ab1ca3b64dfcef209635c44b3444"} Nov 03 08:16:59.260120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:59.260119 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:16:59.276140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:16:59.276079 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-fzvxh" podStartSLOduration=1.7816881260000001 podStartE2EDuration="2.276058517s" totalImagesPullingTime="494.370391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:59.274125875 +0000 UTC m=+1358.655576791" watchObservedRunningTime="2026-11-03 08:16:59.276058517 +0000 UTC m=+1358.657509436" Nov 03 08:17:01.960726 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:01.960682 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:17:06.653956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:06.653919 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28"] Nov 03 08:17:06.654462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:06.654175 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" podUID="ae1c5f9e-1482-4821-9805-ce721e782b7d" containerName="maas-api" containerID="cri-o://32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa" gracePeriod=30 Nov 03 08:17:07.272829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.272420 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:17:07.310274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.310238 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9"] Nov 03 08:17:07.311908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.311879 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ae1c5f9e-1482-4821-9805-ce721e782b7d" containerName="maas-api" Nov 03 08:17:07.311908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.311908 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1c5f9e-1482-4821-9805-ce721e782b7d" containerName="maas-api" Nov 03 08:17:07.312262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.312239 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1c5f9e-1482-4821-9805-ce721e782b7d" containerName="maas-api" Nov 03 08:17:07.336819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.336784 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.349483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.349408 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9"] Nov 03 08:17:07.382927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.382885 5951 generic.go:356] "Generic (PLEG): container finished" podID="ae1c5f9e-1482-4821-9805-ce721e782b7d" containerID="32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa" exitCode=0 Nov 03 08:17:07.383278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.383229 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" event={"ID":"ae1c5f9e-1482-4821-9805-ce721e782b7d","Type":"ContainerDied","Data":"32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa"} Nov 03 08:17:07.383359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.383287 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" event={"ID":"ae1c5f9e-1482-4821-9805-ce721e782b7d","Type":"ContainerDied","Data":"78173504f8dcc6f947fe588d971bb257f723271e57417eb1e69f4698f6f873f2"} Nov 03 08:17:07.383359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.383321 5951 scope.go:122] "RemoveContainer" containerID="32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa" Nov 03 08:17:07.383503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.383485 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28" Nov 03 08:17:07.429338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.429301 5951 scope.go:122] "RemoveContainer" containerID="32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa" Nov 03 08:17:07.430007 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:07.429976 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa\": container with ID starting with 32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa not found: ID does not exist" containerID="32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa" Nov 03 08:17:07.430125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.430019 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa"} err="failed to get container status \"32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa\": rpc error: code = NotFound desc = could not find container \"32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa\": container with ID starting with 32c2bf9926f52c0acdc846217db751b871bdfbc12645310c2faf0ff0719aa6aa not found: ID does not exist" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.446748 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca\") pod \"ae1c5f9e-1482-4821-9805-ce721e782b7d\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447077 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs\") pod \"ae1c5f9e-1482-4821-9805-ce721e782b7d\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447165 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls\") pod \"ae1c5f9e-1482-4821-9805-ce721e782b7d\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447198 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs\" (UniqueName: \"kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs\") pod \"ae1c5f9e-1482-4821-9805-ce721e782b7d\" (UID: \"ae1c5f9e-1482-4821-9805-ce721e782b7d\") " Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447543 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447575 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447679 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447718 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447774 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447813 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447848 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.448677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.447872 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.449426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.448887 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca" pod "ae1c5f9e-1482-4821-9805-ce721e782b7d" (UID: "ae1c5f9e-1482-4821-9805-ce721e782b7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:17:07.449426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.449102 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp89d\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.451222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.451188 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls" pod "ae1c5f9e-1482-4821-9805-ce721e782b7d" (UID: "ae1c5f9e-1482-4821-9805-ce721e782b7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:17:07.451352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.451248 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1c5f9e-1482-4821-9805-ce721e782b7d-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:07.454697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.451877 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs" pod "ae1c5f9e-1482-4821-9805-ce721e782b7d" (UID: "ae1c5f9e-1482-4821-9805-ce721e782b7d"). InnerVolumeSpecName "kube-api-access-nfzjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:07.454697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.451957 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs" pod "ae1c5f9e-1482-4821-9805-ce721e782b7d" (UID: "ae1c5f9e-1482-4821-9805-ce721e782b7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:17:07.552858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.552825 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.552858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.552873 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553138 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.552900 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553138 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.552969 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp89d\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553152 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553194 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553302 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553348 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553403 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553524 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553537 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:07.553606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553583 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1c5f9e-1482-4821-9805-ce721e782b7d-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:07.553606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553597 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553603 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfzjs\" (UniqueName: \"kubernetes.io/projected/ae1c5f9e-1482-4821-9805-ce721e782b7d-kube-api-access-nfzjs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:07.553881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553849 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.553881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553868 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.554001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.553884 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.555712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.555696 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.555776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.555730 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.562501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.562473 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.562760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.562742 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp89d\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d\") pod \"e2e-api-b-228695-openshift-default-547df686cf-ff9t9\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.664816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.664773 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:07.787488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.787459 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28"] Nov 03 08:17:07.813281 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:07.813245 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-228695-65c7dcfbb6-24g28"] Nov 03 08:17:07.930497 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:07.930456 5951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1c5f9e_1482_4821_9805_ce721e782b7d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1c5f9e_1482_4821_9805_ce721e782b7d.slice/crio-78173504f8dcc6f947fe588d971bb257f723271e57417eb1e69f4698f6f873f2\": RecentStats: unable to find data in memory cache]" Nov 03 08:17:08.001683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.001633 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9"] Nov 03 08:17:08.003309 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:17:08.003282 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a76ec39_10a3_4fbd_bdf7_1dac44215c3c.slice/crio-54c3363123f0e089b0e543e7af5a05bad8cd604b21374d9deca9dca012931aed WatchSource:0}: Error finding container 54c3363123f0e089b0e543e7af5a05bad8cd604b21374d9deca9dca012931aed: Status 404 returned error can't find the container with id 54c3363123f0e089b0e543e7af5a05bad8cd604b21374d9deca9dca012931aed Nov 03 08:17:08.006543 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.006488 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:08.006683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.006577 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:08.006683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.006623 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:08.398094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.398016 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" event={"ID":"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c","Type":"ContainerStarted","Data":"089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a"} Nov 03 08:17:08.398313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.398159 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" event={"ID":"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c","Type":"ContainerStarted","Data":"54c3363123f0e089b0e543e7af5a05bad8cd604b21374d9deca9dca012931aed"} Nov 03 08:17:08.450771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.450703 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podStartSLOduration=38.450684221 podStartE2EDuration="38.450684221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:08.444144894 +0000 UTC m=+1367.825595813" watchObservedRunningTime="2026-11-03 08:17:08.450684221 +0000 UTC m=+1367.832135137" Nov 03 08:17:08.666094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.666019 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:08.743206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:08.743168 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1c5f9e-1482-4821-9805-ce721e782b7d" path="/var/lib/kubelet/pods/ae1c5f9e-1482-4821-9805-ce721e782b7d/volumes" Nov 03 08:17:09.667136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:09.667100 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-ff9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 08:17:09.667566 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:09.667164 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 08:17:10.272161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:10.272132 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-fzvxh" Nov 03 08:17:10.665839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:10.665805 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-ff9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 08:17:10.666021 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:10.665872 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 08:17:11.666166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:11.666133 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-ff9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 08:17:11.666619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:11.666183 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 08:17:12.666131 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:12.666096 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-ff9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 08:17:12.666294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:12.666149 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 08:17:13.665862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:13.665829 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-ff9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 08:17:13.666053 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:13.665883 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 08:17:14.666289 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:14.666250 5951 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-ff9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 08:17:14.666738 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:14.666316 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 08:17:15.389081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:15.389048 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9"] Nov 03 08:17:15.389986 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:15.389559 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" containerID="cri-o://089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a" gracePeriod=30 Nov 03 08:17:15.483681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:15.481581 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl"] Nov 03 08:17:15.483681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:15.481899 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" containerID="cri-o://0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe" gracePeriod=30 Nov 03 08:17:21.685441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.685112 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:17:21.768805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.768759 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.769174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.769150 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.771202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.769315 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.771531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.771510 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.771746 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.771727 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.772022 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.772000 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.772302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.772270 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.772474 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.772450 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.772631 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.772614 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket\") pod \"dc41e7dc-d51b-4bc4-a217-26026641f277\" (UID: \"dc41e7dc-d51b-4bc4-a217-26026641f277\") " Nov 03 08:17:21.773112 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.773073 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:17:21.773306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.773238 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.773375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.773340 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.773502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.773471 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.773866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.773844 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.774731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.774458 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.774731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.774486 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.774731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.774502 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc41e7dc-d51b-4bc4-a217-26026641f277-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.774731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.774517 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.774731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.774531 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.777865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.777838 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "kube-api-access-chnpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:21.786400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.786371 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.793391 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.791780 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:17:21.799308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.799277 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token" pod "dc41e7dc-d51b-4bc4-a217-26026641f277" (UID: "dc41e7dc-d51b-4bc4-a217-26026641f277"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:21.815999 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.815961 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw"] Nov 03 08:17:21.820179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.818536 5951 cpu_manager.go:414] "containerMap: removing container" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" Nov 03 08:17:21.820179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.818578 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" Nov 03 08:17:21.827375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.825375 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerName="istio-proxy" Nov 03 08:17:21.837166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.835000 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.837166 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.835505 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:21.838385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.838062 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw"] Nov 03 08:17:21.842304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.842283 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a66d5c-openshift-default-dockercfg-kccfj\"" Nov 03 08:17:21.880256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.880160 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.880906 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.880839 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.881184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.881162 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.881320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.881306 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.881555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.881535 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.881824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.881804 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.882089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.882067 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.882392 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.882373 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.882563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.882546 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d\") pod \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\" (UID: \"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c\") " Nov 03 08:17:21.883683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.883630 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.884055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.884034 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.884264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.884247 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.884449 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.884431 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.884695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.884667 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.884884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.884866 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24xt\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.885169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.885151 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.885333 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.885314 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.885479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.885461 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.885987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.885968 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnpz\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-kube-api-access-chnpz\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.886123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.886109 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.886231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.886218 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.886319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.886306 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc41e7dc-d51b-4bc4-a217-26026641f277-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:21.898689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.884739 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.898689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.885461 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.898689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.885720 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.898689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.897883 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.898689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.897977 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:17:21.910767 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.909191 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:17:21.919560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.919513 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:21.925680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.919716 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:17:21.925680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.925207 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d" pod "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" (UID: "6a76ec39-10a3-4fbd-bdf7-1dac44215c3c"). InnerVolumeSpecName "kube-api-access-gp89d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:21.933795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.933757 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv"] Nov 03 08:17:21.935195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.935168 5951 cpu_manager.go:414] "containerMap: removing container" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" Nov 03 08:17:21.935195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.935193 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" Nov 03 08:17:21.935463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.935449 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerName="istio-proxy" Nov 03 08:17:21.943948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.943906 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.945565 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.945526 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv"] Nov 03 08:17:21.947893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.947871 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-99410b-openshift-default-dockercfg-64wgq\"" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.988906 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.988977 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989033 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24xt\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989085 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989125 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989151 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989188 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989295 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989339 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989408 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989481 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989520 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.989666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.989598 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.990551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990348 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.990551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990401 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:21.990711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990571 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:21.990776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990752 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990862 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990933 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.990953 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d8lj\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992062 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992289 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992512 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992608 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992635 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992671 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992685 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992707 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992732 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992747 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp89d\" (UniqueName: \"kubernetes.io/projected/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-kube-api-access-gp89d\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992762 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.992778 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:22.000697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:21.993306 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.008711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.005211 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.008711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.005568 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.008711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.007675 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24xt\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt\") pod \"e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.094543 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094503 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.094775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094567 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.094775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094661 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.094775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094751 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.094946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094834 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.094946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094910 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.095046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.094976 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d8lj\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.095104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.095048 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.095302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.095280 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.095385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.095338 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.095815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.095686 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.095980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.095922 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.096174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.096153 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.096242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.096213 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.098661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.098619 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.098892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.098872 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.103860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.103835 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.104202 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.104178 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d8lj\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.160113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.160013 5951 generic.go:356] "Generic (PLEG): container finished" podID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" containerID="089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a" exitCode=0 Nov 03 08:17:22.160113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.160072 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" Nov 03 08:17:22.160113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.160096 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" event={"ID":"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c","Type":"ContainerDied","Data":"089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a"} Nov 03 08:17:22.160357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.160148 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9" event={"ID":"6a76ec39-10a3-4fbd-bdf7-1dac44215c3c","Type":"ContainerDied","Data":"54c3363123f0e089b0e543e7af5a05bad8cd604b21374d9deca9dca012931aed"} Nov 03 08:17:22.160357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.160170 5951 scope.go:122] "RemoveContainer" containerID="089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a" Nov 03 08:17:22.165483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.162195 5951 generic.go:356] "Generic (PLEG): container finished" podID="dc41e7dc-d51b-4bc4-a217-26026641f277" containerID="0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe" exitCode=0 Nov 03 08:17:22.165483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.162260 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" Nov 03 08:17:22.165483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.162292 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" event={"ID":"dc41e7dc-d51b-4bc4-a217-26026641f277","Type":"ContainerDied","Data":"0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe"} Nov 03 08:17:22.165483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.162344 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl" event={"ID":"dc41e7dc-d51b-4bc4-a217-26026641f277","Type":"ContainerDied","Data":"c761f12cb09afae6f6333d60884139e94e1a1faa737cf83d7481d5dc41d4e35c"} Nov 03 08:17:22.166206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.166182 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:22.179890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.179414 5951 scope.go:122] "RemoveContainer" containerID="089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a" Nov 03 08:17:22.181547 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:22.180913 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a\": container with ID starting with 089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a not found: ID does not exist" containerID="089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a" Nov 03 08:17:22.181547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.180941 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a"} err="failed to get container status \"089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a\": rpc error: code = NotFound desc = could not find container \"089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a\": container with ID starting with 089c86792a99a7d9fc3dce3e4f18d497429a9aa466fef6e928f1badf2afb2e7a not found: ID does not exist" Nov 03 08:17:22.181547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.180959 5951 scope.go:122] "RemoveContainer" containerID="0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe" Nov 03 08:17:22.208931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.208902 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9"] Nov 03 08:17:22.219139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.219076 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-ff9t9"] Nov 03 08:17:22.229729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.229694 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl"] Nov 03 08:17:22.235964 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.235935 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-prtbl"] Nov 03 08:17:22.245947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.245785 5951 scope.go:122] "RemoveContainer" containerID="0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe" Nov 03 08:17:22.246319 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:22.246292 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe\": container with ID starting with 0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe not found: ID does not exist" containerID="0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe" Nov 03 08:17:22.246434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.246330 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe"} err="failed to get container status \"0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe\": rpc error: code = NotFound desc = could not find container \"0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe\": container with ID starting with 0b22709a2ff2505eb07898a8a147f395fcd00d457a7e15233f4816559a7a68fe not found: ID does not exist" Nov 03 08:17:22.267612 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.267584 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:22.499304 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:17:22.499271 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a351ffb_58a7_464c_a905_bcbfe06847ba.slice/crio-8330da983515ffab758ee90d4fa155d85f5b14583a795a86d414be8ad0348278 WatchSource:0}: Error finding container 8330da983515ffab758ee90d4fa155d85f5b14583a795a86d414be8ad0348278: Status 404 returned error can't find the container with id 8330da983515ffab758ee90d4fa155d85f5b14583a795a86d414be8ad0348278 Nov 03 08:17:22.504253 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.504210 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:22.504351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.504312 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:22.504388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.504360 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:22.515198 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.515102 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw"] Nov 03 08:17:22.648213 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.648184 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv"] Nov 03 08:17:22.655826 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:17:22.655791 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e800ea_7b6e_4a7f_915e_7b45adcae7b2.slice/crio-8175c2493a3be3f4bc0c69a78e164c8b6e4239bb28562c1f1db9156a187d4a9c WatchSource:0}: Error finding container 8175c2493a3be3f4bc0c69a78e164c8b6e4239bb28562c1f1db9156a187d4a9c: Status 404 returned error can't find the container with id 8175c2493a3be3f4bc0c69a78e164c8b6e4239bb28562c1f1db9156a187d4a9c Nov 03 08:17:22.659606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.659445 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:22.659606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.659539 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:22.659606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.659569 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:17:22.830671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.808682 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a76ec39-10a3-4fbd-bdf7-1dac44215c3c" path="/var/lib/kubelet/pods/6a76ec39-10a3-4fbd-bdf7-1dac44215c3c/volumes" Nov 03 08:17:22.830671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:22.809539 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc41e7dc-d51b-4bc4-a217-26026641f277" path="/var/lib/kubelet/pods/dc41e7dc-d51b-4bc4-a217-26026641f277/volumes" Nov 03 08:17:23.179815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.179781 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" event={"ID":"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2","Type":"ContainerStarted","Data":"37612b11f778d315a7646cc0cc7dc3a5f7eb0f8a77036435d83681f29b26a3f4"} Nov 03 08:17:23.180031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.179844 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" event={"ID":"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2","Type":"ContainerStarted","Data":"8175c2493a3be3f4bc0c69a78e164c8b6e4239bb28562c1f1db9156a187d4a9c"} Nov 03 08:17:23.182390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.182040 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" event={"ID":"5a351ffb-58a7-464c-a905-bcbfe06847ba","Type":"ContainerStarted","Data":"1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610"} Nov 03 08:17:23.182390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.182143 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" event={"ID":"5a351ffb-58a7-464c-a905-bcbfe06847ba","Type":"ContainerStarted","Data":"8330da983515ffab758ee90d4fa155d85f5b14583a795a86d414be8ad0348278"} Nov 03 08:17:23.212537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.212484 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podStartSLOduration=5.212467288 podStartE2EDuration="5.212467288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:23.199710751 +0000 UTC m=+1382.581161721" watchObservedRunningTime="2026-11-03 08:17:23.212467288 +0000 UTC m=+1382.593918284" Nov 03 08:17:23.241671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.236803 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podStartSLOduration=7.236783147 podStartE2EDuration="7.236783147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:23.221457627 +0000 UTC m=+1382.602908552" watchObservedRunningTime="2026-11-03 08:17:23.236783147 +0000 UTC m=+1382.618234066" Nov 03 08:17:23.268462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.268422 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:23.270588 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.270559 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:23.270771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:23.270604 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:24.167340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:24.167257 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:24.173699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:24.173602 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:24.197670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:24.197574 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:24.233696 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:24.232382 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:17:24.269677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:24.269138 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:24.269677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:24.269215 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:25.268457 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:25.268421 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:25.269118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:25.269081 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:26.269120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:26.269090 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:26.269674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:26.269138 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:27.268509 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:27.268472 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:27.268819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:27.268786 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:28.276722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:28.273727 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:28.276722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:28.273826 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:29.268661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:29.268612 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 08:17:29.268938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:29.268736 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 08:17:30.042666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:30.037812 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg"] Nov 03 08:17:30.081462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:30.081427 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w"] Nov 03 08:17:30.276380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:30.276336 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:30.278257 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:30.278225 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:30.279365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:30.279343 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:17:31.958679 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:31.958627 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:17:35.038785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.038710 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" podUID="e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" containerName="payload-processing" containerID="cri-o://ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502" gracePeriod=25 Nov 03 08:17:35.084025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.083983 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" podUID="666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" containerName="payload-pre-processing" containerID="cri-o://9e4348fb79ded1f4580966c693fdc992c5aaa449928e0b054f7f49de01d68532" gracePeriod=25 Nov 03 08:17:35.484902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.484864 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:17:35.491899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.491848 5951 generic.go:356] "Generic (PLEG): container finished" podID="e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" containerID="ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502" exitCode=0 Nov 03 08:17:35.492063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.492016 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" event={"ID":"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9","Type":"ContainerDied","Data":"ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502"} Nov 03 08:17:35.492127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.492070 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" event={"ID":"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9","Type":"ContainerDied","Data":"0e40ebd4dbf3cc5bcde90d41cc17e9990fc0f6d5f5d01a6f2be9baa8e6546ab9"} Nov 03 08:17:35.492127 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.492100 5951 scope.go:122] "RemoveContainer" containerID="ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502" Nov 03 08:17:35.492284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.492269 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg" Nov 03 08:17:35.526507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.516026 5951 scope.go:122] "RemoveContainer" containerID="ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502" Nov 03 08:17:35.526507 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:17:35.517686 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502\": container with ID starting with ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502 not found: ID does not exist" containerID="ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502" Nov 03 08:17:35.526507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.517721 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502"} err="failed to get container status \"ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502\": rpc error: code = NotFound desc = could not find container \"ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502\": container with ID starting with ff2b5a99131523b495f37fa0953f7cad410c5278d9987c5c77bb1d8a3f035502 not found: ID does not exist" Nov 03 08:17:35.526507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.517910 5951 generic.go:356] "Generic (PLEG): container finished" podID="666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" containerID="9e4348fb79ded1f4580966c693fdc992c5aaa449928e0b054f7f49de01d68532" exitCode=0 Nov 03 08:17:35.526507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.517986 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" event={"ID":"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad","Type":"ContainerDied","Data":"9e4348fb79ded1f4580966c693fdc992c5aaa449928e0b054f7f49de01d68532"} Nov 03 08:17:35.560713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.558970 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume\") pod \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " Nov 03 08:17:35.560713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.559022 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt\" (UniqueName: \"kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt\") pod \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\" (UID: \"e640e5f0-2fd3-44f5-bd24-1dd75c90bde9\") " Nov 03 08:17:35.560713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.560131 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume" pod "e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" (UID: "e640e5f0-2fd3-44f5-bd24-1dd75c90bde9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:17:35.594705 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.560980 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:35.596340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.596303 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt" pod "e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" (UID: "e640e5f0-2fd3-44f5-bd24-1dd75c90bde9"). InnerVolumeSpecName "kube-api-access-hgbwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:35.618720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.618692 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:17:35.661680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.661630 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw\" (UniqueName: \"kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw\") pod \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " Nov 03 08:17:35.661869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.661795 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume\") pod \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\" (UID: \"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad\") " Nov 03 08:17:35.662238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.662209 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume" pod "666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" (UID: "666fb4d8-b2dd-43ad-8df2-c3821bbc15ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:17:35.662816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.662785 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:35.662816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.662810 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbwt\" (UniqueName: \"kubernetes.io/projected/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9-kube-api-access-hgbwt\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:35.664607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.664582 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw" pod "666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" (UID: "666fb4d8-b2dd-43ad-8df2-c3821bbc15ad"). InnerVolumeSpecName "kube-api-access-85jrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:17:35.824596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.824483 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85jrw\" (UniqueName: \"kubernetes.io/projected/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad-kube-api-access-85jrw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:17:35.860908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.860872 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg"] Nov 03 08:17:35.865381 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:35.865352 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-950e06-9dcfb5cdf-x5swg"] Nov 03 08:17:36.531275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.531234 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" event={"ID":"666fb4d8-b2dd-43ad-8df2-c3821bbc15ad","Type":"ContainerDied","Data":"a87f2a3df50b0c5a44fb1ea39362118483d9673c726a7a8aea680df1819cdee9"} Nov 03 08:17:36.531275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.531265 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w" Nov 03 08:17:36.531833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.531354 5951 scope.go:122] "RemoveContainer" containerID="9e4348fb79ded1f4580966c693fdc992c5aaa449928e0b054f7f49de01d68532" Nov 03 08:17:36.558524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.558494 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w"] Nov 03 08:17:36.565274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.565250 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-950e06-85dc7b464c-7657w"] Nov 03 08:17:36.744051 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.744013 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" path="/var/lib/kubelet/pods/666fb4d8-b2dd-43ad-8df2-c3821bbc15ad/volumes" Nov 03 08:17:36.744595 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:36.744577 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" path="/var/lib/kubelet/pods/e640e5f0-2fd3-44f5-bd24-1dd75c90bde9/volumes" Nov 03 08:17:39.368573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.368533 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r"] Nov 03 08:17:39.369452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.369435 5951 cpu_manager.go:414] "containerMap: removing container" podUID="e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" containerName="payload-processing" Nov 03 08:17:39.369492 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.369454 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" containerName="payload-processing" Nov 03 08:17:39.369532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.369511 5951 cpu_manager.go:414] "containerMap: removing container" podUID="666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" containerName="payload-pre-processing" Nov 03 08:17:39.369532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.369516 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" containerName="payload-pre-processing" Nov 03 08:17:39.369686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.369676 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e640e5f0-2fd3-44f5-bd24-1dd75c90bde9" containerName="payload-processing" Nov 03 08:17:39.369736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.369698 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="666fb4d8-b2dd-43ad-8df2-c3821bbc15ad" containerName="payload-pre-processing" Nov 03 08:17:39.386038 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.385437 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:39.389076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.389046 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r"] Nov 03 08:17:39.389708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.389665 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0c8a52\"" Nov 03 08:17:39.390452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.390430 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0c8a52-dockercfg-2chl9\"" Nov 03 08:17:39.810674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.804368 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:39.810674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.804520 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnxb\" (UniqueName: \"kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb\") pod \"payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:39.907065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.906490 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:39.907500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.907477 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnxb\" (UniqueName: \"kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb\") pod \"payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:39.909562 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.909493 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:39.913663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.910941 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn"] Nov 03 08:17:39.919171 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.918313 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:39.923216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.920838 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn"] Nov 03 08:17:39.923216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:39.922401 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnxb\" (UniqueName: \"kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb\") pod \"payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:40.009918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.009880 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25\") pod \"payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.010300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.010276 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.017130 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.017101 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:40.175692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.154225 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25\") pod \"payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.175692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.154527 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.175692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.155583 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.175692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.172300 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25\") pod \"payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.260230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.260183 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.313504 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:17:40.313460 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb26790_a705_4a6b_88ca_96804ad613ba.slice/crio-94d796009e512aec6c23b9c2f74546991fc24b2683b9e000d8851fe77c7b1b51 WatchSource:0}: Error finding container 94d796009e512aec6c23b9c2f74546991fc24b2683b9e000d8851fe77c7b1b51: Status 404 returned error can't find the container with id 94d796009e512aec6c23b9c2f74546991fc24b2683b9e000d8851fe77c7b1b51 Nov 03 08:17:40.318680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.317959 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r"] Nov 03 08:17:40.561992 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:17:40.561948 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea4e042_0716_42e3_a513_92d6917fb51c.slice/crio-244d309eb70d2c07fe4160818530493c184db7e019fa46db51f9179a6f17206c WatchSource:0}: Error finding container 244d309eb70d2c07fe4160818530493c184db7e019fa46db51f9179a6f17206c: Status 404 returned error can't find the container with id 244d309eb70d2c07fe4160818530493c184db7e019fa46db51f9179a6f17206c Nov 03 08:17:40.563151 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.563124 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn"] Nov 03 08:17:40.895998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.895194 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" event={"ID":"7ea4e042-0716-42e3-a513-92d6917fb51c","Type":"ContainerStarted","Data":"e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9"} Nov 03 08:17:40.895998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.895263 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" event={"ID":"7ea4e042-0716-42e3-a513-92d6917fb51c","Type":"ContainerStarted","Data":"244d309eb70d2c07fe4160818530493c184db7e019fa46db51f9179a6f17206c"} Nov 03 08:17:40.895998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.895294 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:17:40.897446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.897413 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" event={"ID":"7fb26790-a705-4a6b-88ca-96804ad613ba","Type":"ContainerStarted","Data":"5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498"} Nov 03 08:17:40.897583 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.897470 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" event={"ID":"7fb26790-a705-4a6b-88ca-96804ad613ba","Type":"ContainerStarted","Data":"94d796009e512aec6c23b9c2f74546991fc24b2683b9e000d8851fe77c7b1b51"} Nov 03 08:17:40.898007 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.897993 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:40.922335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.922254 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" podStartSLOduration=1.9222323270000001 podStartE2EDuration="1.922232327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:40.917765828 +0000 UTC m=+1400.299216757" watchObservedRunningTime="2026-11-03 08:17:40.922232327 +0000 UTC m=+1400.303683245" Nov 03 08:17:40.941171 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:40.941109 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" podStartSLOduration=1.9410860909999998 podStartE2EDuration="1.941086091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:40.935102911 +0000 UTC m=+1400.316553853" watchObservedRunningTime="2026-11-03 08:17:40.941086091 +0000 UTC m=+1400.322537010" Nov 03 08:17:51.924845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:51.924810 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:17:51.925276 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:17:51.924898 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:18:01.958965 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:18:01.958927 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:18:21.546666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:21.546314 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4"] Nov 03 08:18:21.546666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:21.546548 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" containerID="cri-o://cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036" gracePeriod=30 Nov 03 08:18:26.933815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.933785 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:18:26.964355 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.964324 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv"] Nov 03 08:18:26.965305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.965290 5951 cpu_manager.go:414] "containerMap: removing container" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" Nov 03 08:18:26.965367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.965307 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" Nov 03 08:18:26.965477 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.965468 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" Nov 03 08:18:26.965513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.965487 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" Nov 03 08:18:26.969826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.969802 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:26.974416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.973453 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-616e46-openshift-default-dockercfg-mqp6b\"" Nov 03 08:18:26.982366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:26.982340 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv"] Nov 03 08:18:27.052277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052239 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052425 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052501 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052585 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052543 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052712 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052756 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052802 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052837 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.052939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052855 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:18:27.052939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.052874 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data\") pod \"8e71c051-5971-42b9-80a7-6b85964c8350\" (UID: \"8e71c051-5971-42b9-80a7-6b85964c8350\") " Nov 03 08:18:27.053095 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053072 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:18:27.053165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053146 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053223 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:18:27.053315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053264 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:18:27.053531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053355 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:18:27.053531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053384 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053437 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053485 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053568 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053680 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053824 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5k7\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.053900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053869 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.054015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.053970 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.054082 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.054070 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.054145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.054087 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.054145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.054103 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.054145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.054118 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71c051-5971-42b9-80a7-6b85964c8350-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.054145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.054132 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.055249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.055216 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:18:27.055249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.055218 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:18:27.055537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.055521 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:18:27.055885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.055867 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h" pod "8e71c051-5971-42b9-80a7-6b85964c8350" (UID: "8e71c051-5971-42b9-80a7-6b85964c8350"). InnerVolumeSpecName "kube-api-access-pqg9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:18:27.155669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.155552 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.155669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.155629 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.155875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.155799 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.155875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.155851 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156019 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.155987 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.155999 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156051 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156096 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156147 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156175 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156295 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5k7\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156335 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156511 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156491 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqg9h\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-kube-api-access-pqg9h\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.156623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156514 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71c051-5971-42b9-80a7-6b85964c8350-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.156623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156529 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71c051-5971-42b9-80a7-6b85964c8350-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.156623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156543 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71c051-5971-42b9-80a7-6b85964c8350-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:18:27.156871 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156736 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.156951 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.156929 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.158524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.158500 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.158947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.158931 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.169463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.169429 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5k7\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.172533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.170607 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.299142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.299096 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:27.587183 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:18:27.587155 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77624484_a1cd_47bc_a1f0_9c132a0da250.slice/crio-742de3f7d1096e9ddc3ff675f94ee862db331f0f7bb1878f6c8c85babf3c4637 WatchSource:0}: Error finding container 742de3f7d1096e9ddc3ff675f94ee862db331f0f7bb1878f6c8c85babf3c4637: Status 404 returned error can't find the container with id 742de3f7d1096e9ddc3ff675f94ee862db331f0f7bb1878f6c8c85babf3c4637 Nov 03 08:18:27.588773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.588751 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv"] Nov 03 08:18:27.590487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.590455 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:18:27.590572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.590520 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:18:27.590572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.590547 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:18:27.603793 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.603762 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" event={"ID":"77624484-a1cd-47bc-a1f0-9c132a0da250","Type":"ContainerStarted","Data":"742de3f7d1096e9ddc3ff675f94ee862db331f0f7bb1878f6c8c85babf3c4637"} Nov 03 08:18:27.606244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.606213 5951 generic.go:356] "Generic (PLEG): container finished" podID="8e71c051-5971-42b9-80a7-6b85964c8350" containerID="cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036" exitCode=0 Nov 03 08:18:27.606356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.606292 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" Nov 03 08:18:27.606356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.606324 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" event={"ID":"8e71c051-5971-42b9-80a7-6b85964c8350","Type":"ContainerDied","Data":"cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036"} Nov 03 08:18:27.606471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.606378 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4" event={"ID":"8e71c051-5971-42b9-80a7-6b85964c8350","Type":"ContainerDied","Data":"f1a2bb2076fc3586fe2b613831e5361971eb20a23f0368ff64249138e6a32938"} Nov 03 08:18:27.606471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.606415 5951 scope.go:122] "RemoveContainer" containerID="cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036" Nov 03 08:18:27.620242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.620222 5951 scope.go:122] "RemoveContainer" containerID="d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067" Nov 03 08:18:27.633660 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.633613 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4"] Nov 03 08:18:27.641097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.640626 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-950e06-openshift-default-b9fd9c855-q9gn4"] Nov 03 08:18:27.647872 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.647845 5951 scope.go:122] "RemoveContainer" containerID="cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036" Nov 03 08:18:27.648158 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:18:27.648134 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036\": container with ID starting with cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036 not found: ID does not exist" containerID="cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036" Nov 03 08:18:27.648278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.648163 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036"} err="failed to get container status \"cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036\": rpc error: code = NotFound desc = could not find container \"cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036\": container with ID starting with cd9b935f507814b48db0ba031cd2bb12d8a9543dc41622a610411590ae606036 not found: ID does not exist" Nov 03 08:18:27.648278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.648181 5951 scope.go:122] "RemoveContainer" containerID="d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067" Nov 03 08:18:27.648525 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:18:27.648506 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067\": container with ID starting with d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067 not found: ID does not exist" containerID="d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067" Nov 03 08:18:27.648572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:27.648533 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067"} err="failed to get container status \"d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067\": rpc error: code = NotFound desc = could not find container \"d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067\": container with ID starting with d8bcda9d492ba32a140ba6f11b0b8928f960bd6facf1d64247eb0814e6e4d067 not found: ID does not exist" Nov 03 08:18:28.645471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:28.645429 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" event={"ID":"77624484-a1cd-47bc-a1f0-9c132a0da250","Type":"ContainerStarted","Data":"45405e2a4b8427150c4849c5ee4084c807a9adcf6c499951aa5ccc14f6a6d1dd"} Nov 03 08:18:28.668899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:28.668840 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podStartSLOduration=12.668826438 podStartE2EDuration="12.668826438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:18:28.663308279 +0000 UTC m=+1448.044759225" watchObservedRunningTime="2026-11-03 08:18:28.668826438 +0000 UTC m=+1448.050277424" Nov 03 08:18:28.744126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:28.744092 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" path="/var/lib/kubelet/pods/8e71c051-5971-42b9-80a7-6b85964c8350/volumes" Nov 03 08:18:29.299669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:29.299295 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:29.306678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:29.305234 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 08:18:29.306678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:29.305291 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 08:18:30.305046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:30.305007 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 08:18:30.305629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:30.305597 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 08:18:31.314667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:31.312727 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 08:18:31.314667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:31.312790 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 08:18:31.961933 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:18:31.961888 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:18:32.303668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:32.299469 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 08:18:32.303668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:32.299530 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 08:18:33.314716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:33.308920 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 08:18:33.315463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:33.315424 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 08:18:34.452706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:34.452325 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:34.453884 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:34.453857 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:34.458469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:34.458036 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:18:41.799669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:41.797693 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:18:41.799669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:41.799278 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:18:42.257869 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.255796 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.400821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.400765 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.459103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.459055 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.499675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.496756 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.635668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.598850 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.753707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.711043 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 08:18:42.755085 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.712477 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.755594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.755568 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:18:42.755781 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.755755 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.873813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.805720 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 08:18:42.873813 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.805819 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:42.940819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:42.939794 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.045870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.016733 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.045870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.017923 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.045870 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.040030 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:18:43.086972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.086928 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:18:43.177668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.147093 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.579792 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.579744 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 03 08:18:43.679593 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.679545 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:18:43.682982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.681888 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.684550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.684531 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:18:43.749207 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.749167 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.751941 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.751909 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 08:18:43.763985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.756119 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:18:43.763985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.756179 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:18:43.763985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.757800 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.763985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.758953 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:18:43.797777 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.797747 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:18:43.810910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.809673 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.851525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.851433 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:18:43.872629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:43.871089 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:18:44.234504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:44.201881 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:18:44.234504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:44.223999 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:18:44.288446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:44.288417 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:18:44.288796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:44.288781 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:18:44.339668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:44.339002 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:18:53.399184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:53.399134 5951 generic.go:356] "Generic (PLEG): container finished" podID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerID="1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610" exitCode=0 Nov 03 08:18:53.399798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:53.399215 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" event={"ID":"5a351ffb-58a7-464c-a905-bcbfe06847ba","Type":"ContainerDied","Data":"1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610"} Nov 03 08:18:53.399872 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:53.399837 5951 scope.go:122] "RemoveContainer" containerID="1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610" Nov 03 08:18:53.401995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:53.401963 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:18:53.402122 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:53.402054 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:18:53.402122 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:53.402104 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:18:54.198011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:54.197974 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:18:54.413663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:54.413615 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" event={"ID":"5a351ffb-58a7-464c-a905-bcbfe06847ba","Type":"ContainerStarted","Data":"f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e"} Nov 03 08:18:55.167168 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:55.167134 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:18:55.168867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:55.168841 5951 patch_prober.go:29] interesting pod/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 08:18:55.168988 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:55.168896 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 08:18:56.167314 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:56.167272 5951 patch_prober.go:29] interesting pod/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 08:18:56.167841 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:56.167326 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 08:18:57.166888 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:57.166850 5951 patch_prober.go:29] interesting pod/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 08:18:57.167108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:57.166921 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 08:18:58.174757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:58.168380 5951 patch_prober.go:29] interesting pod/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 08:18:58.174757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:58.168473 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 08:18:59.167064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:59.167032 5951 patch_prober.go:29] interesting pod/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 03 08:18:59.167285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:18:59.167104 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 03 08:19:00.205802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:00.205767 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:19:00.207582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:00.207509 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:19:00.211494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:00.211470 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:19:01.957523 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:19:01.957485 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:19:14.636863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:14.636813 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv"] Nov 03 08:19:14.637408 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:14.637083 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" containerID="cri-o://37612b11f778d315a7646cc0cc7dc3a5f7eb0f8a77036435d83681f29b26a3f4" gracePeriod=30 Nov 03 08:19:15.279942 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:15.279885 5951 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:19:15.280126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:15.279966 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:19:19.926403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:19.926316 5951 generic.go:356] "Generic (PLEG): container finished" podID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerID="37612b11f778d315a7646cc0cc7dc3a5f7eb0f8a77036435d83681f29b26a3f4" exitCode=0 Nov 03 08:19:19.926909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:19.926438 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" event={"ID":"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2","Type":"ContainerDied","Data":"37612b11f778d315a7646cc0cc7dc3a5f7eb0f8a77036435d83681f29b26a3f4"} Nov 03 08:19:19.973432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:19.973400 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:19:20.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.011217 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz"] Nov 03 08:19:20.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.012674 5951 cpu_manager.go:414] "containerMap: removing container" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" Nov 03 08:19:20.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.012701 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" Nov 03 08:19:20.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.012747 5951 cpu_manager.go:414] "containerMap: removing container" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" Nov 03 08:19:20.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.012757 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e71c051-5971-42b9-80a7-6b85964c8350" containerName="istio-proxy" Nov 03 08:19:20.017670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.013036 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" containerName="istio-proxy" Nov 03 08:19:20.036721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.025966 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.050527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.050352 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz"] Nov 03 08:19:20.155211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155177 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.155211 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155216 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.155459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155288 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.155459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155337 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.155459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155401 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.155562 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155476 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155725 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155756 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155773 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155775 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155812 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo\") pod \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\" (UID: \"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2\") " Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.155978 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156015 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156049 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156083 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156166 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156197 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156334 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156483 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69dw\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156663 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156750 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156812 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156870 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.156995 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.156990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.157013 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.158025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.157028 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.158025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.157042 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.158025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.157054 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.158724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.158701 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:19:20.159143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.159123 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "kube-api-access-7d8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:19:20.159250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.159151 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:19:20.161820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.161796 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo" pod "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" (UID: "d7e800ea-7b6e-4a7f-915e-7b45adcae7b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:19:20.258349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258308 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258405 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258468 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258563 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258586 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258778 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258863 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.258890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258861 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.259032 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.258998 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69dw\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.259099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259078 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.259156 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259112 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.259247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259224 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.259394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259374 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.259458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259411 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.259458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259427 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.259458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259441 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d8lj\" (UniqueName: \"kubernetes.io/projected/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-kube-api-access-7d8lj\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.259458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259455 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:19:20.259598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.259532 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.261109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.261087 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.261451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.261432 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.266083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.266059 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.266229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.266206 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69dw\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.359437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.359400 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:20.661177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.661141 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz"] Nov 03 08:19:20.667890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.665869 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:19:20.668110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.668074 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:19:20.668296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.668147 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:19:20.939505 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.939472 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" Nov 03 08:19:20.939505 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.939483 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv" event={"ID":"d7e800ea-7b6e-4a7f-915e-7b45adcae7b2","Type":"ContainerDied","Data":"8175c2493a3be3f4bc0c69a78e164c8b6e4239bb28562c1f1db9156a187d4a9c"} Nov 03 08:19:20.940132 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.939561 5951 scope.go:122] "RemoveContainer" containerID="37612b11f778d315a7646cc0cc7dc3a5f7eb0f8a77036435d83681f29b26a3f4" Nov 03 08:19:20.942976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.942917 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" event={"ID":"7b8bdfd9-02f4-4661-b758-67cba3c395e0","Type":"ContainerStarted","Data":"f8ed48c908418555fbe93a8735346e273ef90bfa0e65d1ae60a9308247817377"} Nov 03 08:19:20.943077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.942976 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" event={"ID":"7b8bdfd9-02f4-4661-b758-67cba3c395e0","Type":"ContainerStarted","Data":"2b57510486c785b941e953b026b70202f0b727fa1cc721767031f89dabaef544"} Nov 03 08:19:20.962276 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:20.962195 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podStartSLOduration=94.962176724 podStartE2EDuration="1m34.962176724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:20.959418427 +0000 UTC m=+1500.340869344" watchObservedRunningTime="2026-11-03 08:19:20.962176724 +0000 UTC m=+1500.343627643" Nov 03 08:19:21.007771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.007734 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv"] Nov 03 08:19:21.014805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.014780 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-qn2fv"] Nov 03 08:19:21.371678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.361740 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:21.371678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.365344 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:21.371678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.365400 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:21.739318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.739241 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:19:21.744458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:21.744437 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:19:22.366601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:22.361039 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:22.366601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:22.361098 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:22.742499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:22.742422 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e800ea-7b6e-4a7f-915e-7b45adcae7b2" path="/var/lib/kubelet/pods/d7e800ea-7b6e-4a7f-915e-7b45adcae7b2/volumes" Nov 03 08:19:23.360072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:23.360034 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:23.360284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:23.360089 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:24.360751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:24.360718 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:24.361155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:24.360806 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:25.360823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:25.360789 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:25.361203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:25.360845 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:26.360285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:26.360250 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:26.360487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:26.360323 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:27.363306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:27.363262 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 08:19:27.363833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:27.363352 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 08:19:28.364911 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:28.364884 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:28.365419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:28.365402 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:28.366225 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:19:28.366207 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:19:31.976350 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:19:31.976309 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:20:01.961729 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:20:01.961686 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:20:31.961713 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:20:31.961623 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:20:44.012441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.012403 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn"] Nov 03 08:20:44.013132 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.012733 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" podUID="6024e94c-640b-4ccf-9e39-dd9bd115ba32" containerName="maas-api" containerID="cri-o://e24789bcb833eff4362724910b31fea73453d55bd2ec0602d19f48eeafe21103" gracePeriod=30 Nov 03 08:20:44.258771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.258739 5951 generic.go:356] "Generic (PLEG): container finished" podID="6024e94c-640b-4ccf-9e39-dd9bd115ba32" containerID="e24789bcb833eff4362724910b31fea73453d55bd2ec0602d19f48eeafe21103" exitCode=0 Nov 03 08:20:44.258962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.258909 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" event={"ID":"6024e94c-640b-4ccf-9e39-dd9bd115ba32","Type":"ContainerDied","Data":"e24789bcb833eff4362724910b31fea73453d55bd2ec0602d19f48eeafe21103"} Nov 03 08:20:44.776968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.776941 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:20:44.823666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.818412 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5"] Nov 03 08:20:44.823666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.821954 5951 cpu_manager.go:414] "containerMap: removing container" podUID="6024e94c-640b-4ccf-9e39-dd9bd115ba32" containerName="maas-api" Nov 03 08:20:44.823666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.821980 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6024e94c-640b-4ccf-9e39-dd9bd115ba32" containerName="maas-api" Nov 03 08:20:44.823666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.822191 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6024e94c-640b-4ccf-9e39-dd9bd115ba32" containerName="maas-api" Nov 03 08:20:44.831029 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.831000 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:44.834663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.833978 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0fc79b-serving-cert\"" Nov 03 08:20:44.835104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.835080 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5"] Nov 03 08:20:44.928675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.926966 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs\") pod \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " Nov 03 08:20:44.928675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.927266 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls\") pod \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " Nov 03 08:20:44.928675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.927317 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca\") pod \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " Nov 03 08:20:44.928675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.927455 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm\" (UniqueName: \"kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm\") pod \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\" (UID: \"6024e94c-640b-4ccf-9e39-dd9bd115ba32\") " Nov 03 08:20:44.933929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.929363 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca" pod "6024e94c-640b-4ccf-9e39-dd9bd115ba32" (UID: "6024e94c-640b-4ccf-9e39-dd9bd115ba32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:20:44.933929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.930413 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:44.933929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.930767 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6d9z\" (UniqueName: \"kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:44.933929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.931379 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:44.933929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.931532 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:44.949481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.941293 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls" pod "6024e94c-640b-4ccf-9e39-dd9bd115ba32" (UID: "6024e94c-640b-4ccf-9e39-dd9bd115ba32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:20:44.949481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.941711 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6024e94c-640b-4ccf-9e39-dd9bd115ba32-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:20:44.971286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.959905 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm" pod "6024e94c-640b-4ccf-9e39-dd9bd115ba32" (UID: "6024e94c-640b-4ccf-9e39-dd9bd115ba32"). InnerVolumeSpecName "kube-api-access-pspkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:20:44.985522 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:44.977200 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs" pod "6024e94c-640b-4ccf-9e39-dd9bd115ba32" (UID: "6024e94c-640b-4ccf-9e39-dd9bd115ba32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:20:45.043950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.043907 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.044451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044042 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6d9z\" (UniqueName: \"kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.044451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044147 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.044451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044185 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.044729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044636 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:20:45.044729 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044723 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pspkm\" (UniqueName: \"kubernetes.io/projected/6024e94c-640b-4ccf-9e39-dd9bd115ba32-kube-api-access-pspkm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:20:45.044865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044739 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6024e94c-640b-4ccf-9e39-dd9bd115ba32-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:20:45.044958 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.044930 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.046965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.046940 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.047344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.047322 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.051636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.051613 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6d9z\" (UniqueName: \"kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z\") pod \"maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.153872 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.153838 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:45.273377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.273351 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" Nov 03 08:20:45.273527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.273415 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn" event={"ID":"6024e94c-640b-4ccf-9e39-dd9bd115ba32","Type":"ContainerDied","Data":"21699ea3576808ac85f5ea6604030efc119adefb31b7e7f294497acce9a12cc7"} Nov 03 08:20:45.273527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.273478 5951 scope.go:122] "RemoveContainer" containerID="e24789bcb833eff4362724910b31fea73453d55bd2ec0602d19f48eeafe21103" Nov 03 08:20:45.307758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.307725 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn"] Nov 03 08:20:45.311563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.311536 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27c674-c78f5fd9-8t9pn"] Nov 03 08:20:45.463340 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:20:45.463272 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcdb196_e296_4f90_95c7_fb1136ac2885.slice/crio-0b78893f60adf1c98ae48340da9fe21b92ee71e3c35581cfc1b6ee147ea42a10 WatchSource:0}: Error finding container 0b78893f60adf1c98ae48340da9fe21b92ee71e3c35581cfc1b6ee147ea42a10: Status 404 returned error can't find the container with id 0b78893f60adf1c98ae48340da9fe21b92ee71e3c35581cfc1b6ee147ea42a10 Nov 03 08:20:45.464773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.464741 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:20:45.466849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:45.466823 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5"] Nov 03 08:20:46.285744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:46.285706 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" event={"ID":"cdcdb196-e296-4f90-95c7-fb1136ac2885","Type":"ContainerStarted","Data":"0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be"} Nov 03 08:20:46.286247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:46.285773 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" event={"ID":"cdcdb196-e296-4f90-95c7-fb1136ac2885","Type":"ContainerStarted","Data":"0b78893f60adf1c98ae48340da9fe21b92ee71e3c35581cfc1b6ee147ea42a10"} Nov 03 08:20:46.286247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:46.285802 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:20:46.309594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:46.309492 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" podStartSLOduration=60.73023994 podStartE2EDuration="1m1.309476662s" totalImagesPullingTime="579.236722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:20:46.304717701 +0000 UTC m=+1585.686168618" watchObservedRunningTime="2026-11-03 08:20:46.309476662 +0000 UTC m=+1585.690927578" Nov 03 08:20:46.747637 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:46.747548 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6024e94c-640b-4ccf-9e39-dd9bd115ba32" path="/var/lib/kubelet/pods/6024e94c-640b-4ccf-9e39-dd9bd115ba32/volumes" Nov 03 08:20:52.311297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:20:52.311263 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:21:01.958089 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:01.958047 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:21:17.306707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:17.306133 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9"] Nov 03 08:21:17.306707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:17.306532 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" containerID="cri-o://8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940" gracePeriod=30 Nov 03 08:21:17.429254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:17.429213 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5"] Nov 03 08:21:17.429544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:17.429475 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" containerID="cri-o://355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6" gracePeriod=30 Nov 03 08:21:17.440844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:17.440782 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75"] Nov 03 08:21:17.441162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:17.441095 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" containerID="cri-o://2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc" gracePeriod=30 Nov 03 08:21:22.685693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.684944 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:21:22.766325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766286 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766337 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766422 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766465 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766502 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766676 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766755 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.766972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766913 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.767132 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.766990 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data\") pod \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\" (UID: \"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a\") " Nov 03 08:21:22.768205 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.768176 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:22.768491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.768443 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:22.768688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.768628 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:22.771562 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.771518 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:21:22.773826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.773687 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:22.774117 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.774079 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:22.775826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.775728 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "kube-api-access-54l5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:22.775826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.775798 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:22.782452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.782422 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token" pod "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" (UID: "4c83392b-7a26-4cfc-9998-a8a8a5ccf89a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:22.802665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.792924 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb"] Nov 03 08:21:22.802665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.800213 5951 cpu_manager.go:414] "containerMap: removing container" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" Nov 03 08:21:22.802665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.800232 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" Nov 03 08:21:22.802665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.800417 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerName="istio-proxy" Nov 03 08:21:22.813563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.804823 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.827679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.817216 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d168e-openshift-default-dockercfg-pr6zq\"" Nov 03 08:21:22.847375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.847224 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb"] Nov 03 08:21:22.888083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.887086 5951 generic.go:356] "Generic (PLEG): container finished" podID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerID="2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc" exitCode=0 Nov 03 08:21:22.888083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.887510 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" event={"ID":"391919b4-ac7b-4656-ab8c-9650666c3d64","Type":"ContainerDied","Data":"2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc"} Nov 03 08:21:22.889776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.889741 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.889917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.889866 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.889996 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.889931 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.890064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.890024 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.890126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.890075 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.890126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.890108 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.890243 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.890136 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x5mj\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.890557 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.890459 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.890687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.890596 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.891040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891015 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891053 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54l5c\" (UniqueName: \"kubernetes.io/projected/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-kube-api-access-54l5c\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891093 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891111 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891125 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891162 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891180 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891196 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891212 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:22.891552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.891489 5951 generic.go:356] "Generic (PLEG): container finished" podID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" containerID="8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940" exitCode=0 Nov 03 08:21:22.892052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.892036 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" Nov 03 08:21:22.913805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.892050 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" event={"ID":"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a","Type":"ContainerDied","Data":"8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940"} Nov 03 08:21:22.914076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.914055 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9" event={"ID":"4c83392b-7a26-4cfc-9998-a8a8a5ccf89a","Type":"ContainerDied","Data":"399f3b5814defca8348b07893c0b70ea72679c15af3e03de93434cf580cffe04"} Nov 03 08:21:22.914177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.914166 5951 scope.go:122] "RemoveContainer" containerID="8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940" Nov 03 08:21:22.921516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.921483 5951 generic.go:356] "Generic (PLEG): container finished" podID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerID="355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6" exitCode=0 Nov 03 08:21:22.921852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.921809 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" event={"ID":"36c490ea-fbef-4135-a1ad-5e67d92bd9bc","Type":"ContainerDied","Data":"355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6"} Nov 03 08:21:22.951683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.951635 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:21:22.959006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.958981 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9"] Nov 03 08:21:22.976128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.976083 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-l77w9"] Nov 03 08:21:22.977921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.977898 5951 scope.go:122] "RemoveContainer" containerID="8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940" Nov 03 08:21:22.979063 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:22.978978 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940\": container with ID starting with 8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940 not found: ID does not exist" containerID="8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940" Nov 03 08:21:22.979179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.979075 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940"} err="failed to get container status \"8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940\": rpc error: code = NotFound desc = could not find container \"8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940\": container with ID starting with 8ec398233313665509ecf9fdd907b3405993189614a0b50e912f1c6ce4ad1940 not found: ID does not exist" Nov 03 08:21:22.995177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.995143 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.995422 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.995392 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.995563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.995549 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.995707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.995693 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.995959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.995927 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.996301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.996286 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.996438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.996424 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.996740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.996722 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.996940 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.996925 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket\") pod \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\" (UID: \"36c490ea-fbef-4135-a1ad-5e67d92bd9bc\") " Nov 03 08:21:22.997824 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.997573 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.997966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.997852 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998003 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998154 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998088 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998217 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998307 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998294 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998662 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998324 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998662 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998362 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x5mj\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.998662 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.998508 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:22.999600 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.999292 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.000059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:22.999740 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.000280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.000118 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.000280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.000223 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.000556 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.000402 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.019909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.018810 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.020482 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.020119 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.020482 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.020288 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.020482 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.020435 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:23.021866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.020800 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:23.021866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.020859 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:21:23.023959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.023745 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn"] Nov 03 08:21:23.026573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.025855 5951 cpu_manager.go:414] "containerMap: removing container" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" Nov 03 08:21:23.026573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.025877 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" Nov 03 08:21:23.026573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.026089 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" containerName="istio-proxy" Nov 03 08:21:23.026573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.026111 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.029369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.027804 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.029369 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.028325 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.033200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.032851 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x5mj\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.033200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.032859 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:21:23.037022 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.035782 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn"] Nov 03 08:21:23.037022 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.035886 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.041789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.041424 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.041789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.041548 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr" pod "36c490ea-fbef-4135-a1ad-5e67d92bd9bc" (UID: "36c490ea-fbef-4135-a1ad-5e67d92bd9bc"). InnerVolumeSpecName "kube-api-access-bbgtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:23.041961 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:23.041871 5951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c83392b_7a26_4cfc_9998_a8a8a5ccf89a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c83392b_7a26_4cfc_9998_a8a8a5ccf89a.slice/crio-399f3b5814defca8348b07893c0b70ea72679c15af3e03de93434cf580cffe04\": RecentStats: unable to find data in memory cache]" Nov 03 08:21:23.044247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.044197 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token\") pod \"e2e-inf-b-0d168e-openshift-default-7b88884569-98phb\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.070741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.070712 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2"] Nov 03 08:21:23.071822 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.071802 5951 cpu_manager.go:414] "containerMap: removing container" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" Nov 03 08:21:23.071944 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.071934 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" Nov 03 08:21:23.072180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.072170 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" containerName="istio-proxy" Nov 03 08:21:23.078300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.078264 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.101365 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.101321 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.101718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.101697 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.101881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.101867 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.102002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.101990 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.102242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.102227 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.102372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.102358 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.102528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.102512 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.102777 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.102758 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.102919 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.102903 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token\") pod \"391919b4-ac7b-4656-ab8c-9650666c3d64\" (UID: \"391919b4-ac7b-4656-ab8c-9650666c3d64\") " Nov 03 08:21:23.103309 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.102979 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:23.103695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103425 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.103695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103464 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.103695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103499 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.103695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103529 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.103695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103616 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103845 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrss\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103903 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b27z\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103916 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.104165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.103939 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104185 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104224 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104269 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104301 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104323 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104358 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104436 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104450 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.104693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104486 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104522 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.104693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104616 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.104693 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104626 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391919b4-ac7b-4656-ab8c-9650666c3d64-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104637 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104759 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104773 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104785 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104799 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104811 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104826 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbgtr\" (UniqueName: \"kubernetes.io/projected/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-kube-api-access-bbgtr\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104841 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.104854 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c490ea-fbef-4135-a1ad-5e67d92bd9bc-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.105447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.105323 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.106701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.105588 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.106701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.105792 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.106701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.106503 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "kube-api-access-mdrdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:23.107089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.107051 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:21:23.109359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.109338 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:23.111182 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.111099 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2"] Nov 03 08:21:23.114008 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.113976 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy" pod "391919b4-ac7b-4656-ab8c-9650666c3d64" (UID: "391919b4-ac7b-4656-ab8c-9650666c3d64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:21:23.216052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.215896 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.216556 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.216527 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrss\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.216756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.216737 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b27z\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.217005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.216905 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.218038 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.217081 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.218207 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.218187 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.219491 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.219469 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.219690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.219671 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.219843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.219824 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.219974 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.219958 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.220108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220080 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.220222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220207 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.220297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220235 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.220357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220295 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.220357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220335 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.220464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220363 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.220464 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220440 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.220568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220468 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.220568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220502 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.220568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220533 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.220797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220717 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.220797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220737 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.220797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220751 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrdv\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-kube-api-access-mdrdv\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.220797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220767 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.220797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220782 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.220797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220795 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.221111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.220811 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391919b4-ac7b-4656-ab8c-9650666c3d64-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:23.221111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.221085 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.221728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.221707 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:23.222080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.222056 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.222835 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.222483 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.223317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.223216 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.223317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.218738 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.223469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.223314 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.224203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.223555 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.224203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.223803 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.224203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.224140 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.229276 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.229230 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.236160 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.236112 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.238803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.238750 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.240313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.240285 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrss\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.245098 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.245037 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.251799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.251753 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b27z\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.261800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.260913 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo\") pod \"e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.365257 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.365001 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:23.414797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.414758 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:23.603441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.603379 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb"] Nov 03 08:21:23.640818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.640081 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.640818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.640177 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.640818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.640221 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.743081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.743053 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn"] Nov 03 08:21:23.743893 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:21:23.743870 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976598fb_dba9_4fb3_a581_7c780f7c56cc.slice/crio-c3fc4b66b87521da5fbc21b1b7051d2e5fe04ca4aea64ef84a921ad55c37d0eb WatchSource:0}: Error finding container c3fc4b66b87521da5fbc21b1b7051d2e5fe04ca4aea64ef84a921ad55c37d0eb: Status 404 returned error can't find the container with id c3fc4b66b87521da5fbc21b1b7051d2e5fe04ca4aea64ef84a921ad55c37d0eb Nov 03 08:21:23.747573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.747539 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.747744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.747599 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.747744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.747632 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.824326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.820324 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2"] Nov 03 08:21:23.851857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.851818 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.851987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.851918 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:23.851987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:23.851970 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:21:24.037453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.037403 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" event={"ID":"0f909765-4f94-4242-bdda-9f9d8f345397","Type":"ContainerStarted","Data":"4e8373b92a5e302296ddf18767c0403f973ad41008b2e71cdb0b8bfd72a18fbf"} Nov 03 08:21:24.055677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.055621 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" event={"ID":"36c490ea-fbef-4135-a1ad-5e67d92bd9bc","Type":"ContainerDied","Data":"6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704"} Nov 03 08:21:24.055949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.055930 5951 scope.go:122] "RemoveContainer" containerID="355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6" Nov 03 08:21:24.056224 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.056204 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5" Nov 03 08:21:24.098351 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.098115 5951 scope.go:122] "RemoveContainer" containerID="355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6" Nov 03 08:21:24.101260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.101231 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" event={"ID":"11d81aa6-a90e-4457-8dd9-41a95c107643","Type":"ContainerStarted","Data":"c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f"} Nov 03 08:21:24.103408 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.103384 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" event={"ID":"11d81aa6-a90e-4457-8dd9-41a95c107643","Type":"ContainerStarted","Data":"19005239c66489c14d9de2630a4c88db833361bee01d746093109abbfe0c2cec"} Nov 03 08:21:24.116830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.116714 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" event={"ID":"391919b4-ac7b-4656-ab8c-9650666c3d64","Type":"ContainerDied","Data":"2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac"} Nov 03 08:21:24.119743 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.117054 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75" Nov 03 08:21:24.141630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.141570 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podStartSLOduration=28.141554436 podStartE2EDuration="28.141554436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:24.135146974 +0000 UTC m=+1623.516597902" watchObservedRunningTime="2026-11-03 08:21:24.141554436 +0000 UTC m=+1623.523005353" Nov 03 08:21:24.142005 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.141973 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" event={"ID":"976598fb-dba9-4fb3-a581-7c780f7c56cc","Type":"ContainerStarted","Data":"71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a"} Nov 03 08:21:24.142220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.142196 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" event={"ID":"976598fb-dba9-4fb3-a581-7c780f7c56cc","Type":"ContainerStarted","Data":"c3fc4b66b87521da5fbc21b1b7051d2e5fe04ca4aea64ef84a921ad55c37d0eb"} Nov 03 08:21:24.172524 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:24.171441 5951 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5_openshift-ingress_36c490ea-fbef-4135-a1ad-5e67d92bd9bc_0 in pod sandbox 6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704 from index: no such id: '355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6'" containerID="355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6" Nov 03 08:21:24.172524 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:24.171492 5951 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5_openshift-ingress_36c490ea-fbef-4135-a1ad-5e67d92bd9bc_0 in pod sandbox 6d5f4959c17f6ca1154ff19c0551cc4d8e7eb75205ce4a01a61b19ea1ab2d704 from index: no such id: '355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6'" containerID="355fc3ee46bdb06eb32d177bb70e9bbac3e1384658268a108f911e5f1421ead6" Nov 03 08:21:24.172524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.171539 5951 scope.go:122] "RemoveContainer" containerID="2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc" Nov 03 08:21:24.174087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.174068 5951 scope.go:122] "RemoveContainer" containerID="2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc" Nov 03 08:21:24.176808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.176453 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podStartSLOduration=144.176439966 podStartE2EDuration="2m24.176439966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:24.169202089 +0000 UTC m=+1623.550653040" watchObservedRunningTime="2026-11-03 08:21:24.176439966 +0000 UTC m=+1623.557890882" Nov 03 08:21:24.194176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.194146 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5"] Nov 03 08:21:24.205630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.205604 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-whnt5"] Nov 03 08:21:24.229399 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:24.229345 5951 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75_openshift-ingress_391919b4-ac7b-4656-ab8c-9650666c3d64_0 in pod sandbox 2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac from index: no such id: '2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc'" containerID="2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc" Nov 03 08:21:24.229579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.229399 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75_openshift-ingress_391919b4-ac7b-4656-ab8c-9650666c3d64_0 in pod sandbox 2f4edf9f940ad12c934a2500de845b1a4fb23b533f73887ed5023b0b83d44aac from index: no such id: '2c28e36cc73a3336d5adbd2e34e785a0382d027e688c63904a9d1c5a3e4267bc'" Nov 03 08:21:24.230332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.230309 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:24.234516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.234490 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:21:24.247024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.246989 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:21:24.247525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.238306 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75"] Nov 03 08:21:24.247730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.247714 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-w5l75"] Nov 03 08:21:24.367387 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.367324 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:24.370214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.369893 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:24.370214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.369950 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:24.742806 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.742735 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c490ea-fbef-4135-a1ad-5e67d92bd9bc" path="/var/lib/kubelet/pods/36c490ea-fbef-4135-a1ad-5e67d92bd9bc/volumes" Nov 03 08:21:24.744350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.744316 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391919b4-ac7b-4656-ab8c-9650666c3d64" path="/var/lib/kubelet/pods/391919b4-ac7b-4656-ab8c-9650666c3d64/volumes" Nov 03 08:21:24.744923 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:24.744903 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c83392b-7a26-4cfc-9998-a8a8a5ccf89a" path="/var/lib/kubelet/pods/4c83392b-7a26-4cfc-9998-a8a8a5ccf89a/volumes" Nov 03 08:21:25.160945 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.160904 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" event={"ID":"0f909765-4f94-4242-bdda-9f9d8f345397","Type":"ContainerStarted","Data":"0e08cc96ea919fa77fa41dd14c77d5a9a4bba2d73223b63933586dca6bce25d1"} Nov 03 08:21:25.211706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.206618 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podStartSLOduration=219.206596899 podStartE2EDuration="3m39.206596899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:25.186726964 +0000 UTC m=+1624.568177884" watchObservedRunningTime="2026-11-03 08:21:25.206596899 +0000 UTC m=+1624.588047817" Nov 03 08:21:25.254072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.254037 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:25.381086 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.380554 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:25.381086 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.380612 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:25.444728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.433513 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:25.445581 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.445544 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:25.447569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:25.447535 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:26.178440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:26.178399 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:26.179524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:26.179505 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:21:26.389244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:26.389211 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:26.389504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:26.389480 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:26.435025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:26.434932 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:26.466668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:26.464838 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:27.369739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:27.368112 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:27.369739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:27.368189 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:27.430268 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:27.430232 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:27.430552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:27.430525 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:28.526547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:28.520559 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:28.526547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:28.520656 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:28.532606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:28.531157 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:28.532606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:28.531258 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:29.372990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:29.372200 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:29.372990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:29.372258 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:29.421803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:29.421766 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:29.422623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:29.422588 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:30.372668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:30.372081 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:30.372668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:30.372160 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:30.446596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:30.435333 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:30.446596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:30.435406 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:31.373709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:31.367093 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 08:21:31.373709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:31.367174 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 08:21:31.416665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:31.415777 5951 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 03 08:21:31.416665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:31.415873 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 03 08:21:31.964902 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:31.964776 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:21:32.380318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:32.380277 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:32.395425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:32.394693 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:32.402897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:32.402829 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:21:32.422507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:32.422477 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:32.422929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:32.422893 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:32.423699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:32.423677 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:21:43.272714 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.272666 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq"] Nov 03 08:21:43.281575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.281535 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq"] Nov 03 08:21:43.281795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.281713 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.285833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.285789 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\"" Nov 03 08:21:43.285975 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.285877 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 08:21:43.495475 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.495430 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.495708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.495482 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zx4\" (UniqueName: \"kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.495708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.495544 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.598152 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.598098 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.598152 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.598151 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zx4\" (UniqueName: \"kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.598436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.598193 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.599196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.599169 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.614106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.614077 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.617034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.617005 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zx4\" (UniqueName: \"kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4\") pod \"maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.628048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.628026 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:43.792743 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.792685 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z"] Nov 03 08:21:43.823683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.807872 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z"] Nov 03 08:21:43.823683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.807995 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:43.905701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.905583 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:43.905853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.905781 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6j6\" (UniqueName: \"kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:43.905853 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.905833 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:43.928549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:43.928518 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq"] Nov 03 08:21:43.931380 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:21:43.931349 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec46b2_7625_4160_9529_b0f6f780b7fb.slice/crio-6eb2dd00fbab47a34edcf8eb8c30212d13cfc0efef0031c364cb31e16898a67a WatchSource:0}: Error finding container 6eb2dd00fbab47a34edcf8eb8c30212d13cfc0efef0031c364cb31e16898a67a: Status 404 returned error can't find the container with id 6eb2dd00fbab47a34edcf8eb8c30212d13cfc0efef0031c364cb31e16898a67a Nov 03 08:21:44.014669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.012487 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.014669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.012599 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6j6\" (UniqueName: \"kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.014669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.012665 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.019668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.016299 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.044867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.042781 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.049502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.049472 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6j6\" (UniqueName: \"kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6\") pod \"maas-api-revoke-sub-6e7ab602-ct76z\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.138528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.138494 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:44.440789 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:21:44.440759 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96897ed3_96a4_46a1_87aa_99333dae25eb.slice/crio-f2f3b5bee3670e777c198281337b8986a3a83f81e52bdb2de9d4aa4faf640752 WatchSource:0}: Error finding container f2f3b5bee3670e777c198281337b8986a3a83f81e52bdb2de9d4aa4faf640752: Status 404 returned error can't find the container with id f2f3b5bee3670e777c198281337b8986a3a83f81e52bdb2de9d4aa4faf640752 Nov 03 08:21:44.440789 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.440776 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z"] Nov 03 08:21:44.981670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.980872 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" event={"ID":"98ec46b2-7625-4160-9529-b0f6f780b7fb","Type":"ContainerStarted","Data":"d4f42ca9f59c04c08bc9dc02b5835d90badbe1ed56343e6dcada136b9411763b"} Nov 03 08:21:44.981670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:44.980946 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" event={"ID":"98ec46b2-7625-4160-9529-b0f6f780b7fb","Type":"ContainerStarted","Data":"6eb2dd00fbab47a34edcf8eb8c30212d13cfc0efef0031c364cb31e16898a67a"} Nov 03 08:21:45.000733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:45.000580 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" event={"ID":"96897ed3-96a4-46a1-87aa-99333dae25eb","Type":"ContainerStarted","Data":"f2f3b5bee3670e777c198281337b8986a3a83f81e52bdb2de9d4aa4faf640752"} Nov 03 08:21:45.009667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:45.007623 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" podStartSLOduration=2.007604594 podStartE2EDuration="2.007604594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:45.001239362 +0000 UTC m=+1644.382690278" watchObservedRunningTime="2026-11-03 08:21:45.007604594 +0000 UTC m=+1644.389055512" Nov 03 08:21:45.263059 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:45.262985 5951 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec46b2_7625_4160_9529_b0f6f780b7fb.slice/crio-conmon-d4f42ca9f59c04c08bc9dc02b5835d90badbe1ed56343e6dcada136b9411763b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec46b2_7625_4160_9529_b0f6f780b7fb.slice/crio-d4f42ca9f59c04c08bc9dc02b5835d90badbe1ed56343e6dcada136b9411763b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 08:21:46.015048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:46.014944 5951 generic.go:356] "Generic (PLEG): container finished" podID="98ec46b2-7625-4160-9529-b0f6f780b7fb" containerID="d4f42ca9f59c04c08bc9dc02b5835d90badbe1ed56343e6dcada136b9411763b" exitCode=0 Nov 03 08:21:46.015048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:46.015027 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" event={"ID":"98ec46b2-7625-4160-9529-b0f6f780b7fb","Type":"ContainerDied","Data":"d4f42ca9f59c04c08bc9dc02b5835d90badbe1ed56343e6dcada136b9411763b"} Nov 03 08:21:46.017316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:46.017237 5951 generic.go:356] "Generic (PLEG): container finished" podID="96897ed3-96a4-46a1-87aa-99333dae25eb" containerID="8843b5648ecd67b3f96b14ac21953ace83040bee4df544d83bafd2baa43cfdae" exitCode=0 Nov 03 08:21:46.017451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:46.017329 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" event={"ID":"96897ed3-96a4-46a1-87aa-99333dae25eb","Type":"ContainerDied","Data":"8843b5648ecd67b3f96b14ac21953ace83040bee4df544d83bafd2baa43cfdae"} Nov 03 08:21:47.335798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.335773 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:47.342232 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.342205 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:47.379344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.378859 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6\" (UniqueName: \"kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6\") pod \"96897ed3-96a4-46a1-87aa-99333dae25eb\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " Nov 03 08:21:47.379344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.378930 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca\") pod \"98ec46b2-7625-4160-9529-b0f6f780b7fb\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " Nov 03 08:21:47.379344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.378995 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca\") pod \"98ec46b2-7625-4160-9529-b0f6f780b7fb\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " Nov 03 08:21:47.379344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.379065 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca\") pod \"96897ed3-96a4-46a1-87aa-99333dae25eb\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " Nov 03 08:21:47.379344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.379113 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4\" (UniqueName: \"kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4\") pod \"98ec46b2-7625-4160-9529-b0f6f780b7fb\" (UID: \"98ec46b2-7625-4160-9529-b0f6f780b7fb\") " Nov 03 08:21:47.379344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.379169 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca\") pod \"96897ed3-96a4-46a1-87aa-99333dae25eb\" (UID: \"96897ed3-96a4-46a1-87aa-99333dae25eb\") " Nov 03 08:21:47.384179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.384152 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca" pod "98ec46b2-7625-4160-9529-b0f6f780b7fb" (UID: "98ec46b2-7625-4160-9529-b0f6f780b7fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:47.385363 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.385316 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca" pod "98ec46b2-7625-4160-9529-b0f6f780b7fb" (UID: "98ec46b2-7625-4160-9529-b0f6f780b7fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:21:47.386065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.386043 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca" pod "96897ed3-96a4-46a1-87aa-99333dae25eb" (UID: "96897ed3-96a4-46a1-87aa-99333dae25eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:47.387093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.387061 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca" pod "96897ed3-96a4-46a1-87aa-99333dae25eb" (UID: "96897ed3-96a4-46a1-87aa-99333dae25eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:21:47.387368 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.387338 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6" pod "96897ed3-96a4-46a1-87aa-99333dae25eb" (UID: "96897ed3-96a4-46a1-87aa-99333dae25eb"). InnerVolumeSpecName "kube-api-access-kq6j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:47.389538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.389505 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4" pod "98ec46b2-7625-4160-9529-b0f6f780b7fb" (UID: "98ec46b2-7625-4160-9529-b0f6f780b7fb"). InnerVolumeSpecName "kube-api-access-x9zx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:47.482501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.482455 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6j6\" (UniqueName: \"kubernetes.io/projected/96897ed3-96a4-46a1-87aa-99333dae25eb-kube-api-access-kq6j6\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:47.482501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.482494 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-openshift-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:47.482754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.482510 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec46b2-7625-4160-9529-b0f6f780b7fb-maas-api-xks-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:47.482754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.482524 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-xks-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:47.482754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.482535 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zx4\" (UniqueName: \"kubernetes.io/projected/98ec46b2-7625-4160-9529-b0f6f780b7fb-kube-api-access-x9zx4\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:47.482754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:47.482547 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96897ed3-96a4-46a1-87aa-99333dae25eb-maas-api-openshift-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:48.043402 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:48.043363 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" Nov 03 08:21:48.043591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:48.043366 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq" event={"ID":"98ec46b2-7625-4160-9529-b0f6f780b7fb","Type":"ContainerDied","Data":"6eb2dd00fbab47a34edcf8eb8c30212d13cfc0efef0031c364cb31e16898a67a"} Nov 03 08:21:48.043591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:48.043542 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb2dd00fbab47a34edcf8eb8c30212d13cfc0efef0031c364cb31e16898a67a" Nov 03 08:21:48.044980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:48.044956 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" Nov 03 08:21:48.045104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:48.045002 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z" event={"ID":"96897ed3-96a4-46a1-87aa-99333dae25eb","Type":"ContainerDied","Data":"f2f3b5bee3670e777c198281337b8986a3a83f81e52bdb2de9d4aa4faf640752"} Nov 03 08:21:48.045104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:48.045040 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2f3b5bee3670e777c198281337b8986a3a83f81e52bdb2de9d4aa4faf640752" Nov 03 08:21:52.551892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:52.551850 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn"] Nov 03 08:21:52.569529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:52.569489 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r"] Nov 03 08:21:57.552796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:57.552759 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" podUID="7ea4e042-0716-42e3-a513-92d6917fb51c" containerName="payload-processing" containerID="cri-o://e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9" gracePeriod=25 Nov 03 08:21:57.570552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:57.570515 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" podUID="7fb26790-a705-4a6b-88ca-96804ad613ba" containerName="payload-pre-processing" containerID="cri-o://5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498" gracePeriod=25 Nov 03 08:21:57.927623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:57.927585 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:21:58.006624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.006432 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume\") pod \"7ea4e042-0716-42e3-a513-92d6917fb51c\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " Nov 03 08:21:58.017814 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.010831 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume" pod "7ea4e042-0716-42e3-a513-92d6917fb51c" (UID: "7ea4e042-0716-42e3-a513-92d6917fb51c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:58.064755 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.064712 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:21:58.112815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.112738 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25\") pod \"7ea4e042-0716-42e3-a513-92d6917fb51c\" (UID: \"7ea4e042-0716-42e3-a513-92d6917fb51c\") " Nov 03 08:21:58.112984 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.112875 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb\" (UniqueName: \"kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb\") pod \"7fb26790-a705-4a6b-88ca-96804ad613ba\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " Nov 03 08:21:58.113176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.113155 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume\") pod \"7fb26790-a705-4a6b-88ca-96804ad613ba\" (UID: \"7fb26790-a705-4a6b-88ca-96804ad613ba\") " Nov 03 08:21:58.113745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.113721 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume" pod "7fb26790-a705-4a6b-88ca-96804ad613ba" (UID: "7fb26790-a705-4a6b-88ca-96804ad613ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:21:58.115010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.114387 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25" pod "7ea4e042-0716-42e3-a513-92d6917fb51c" (UID: "7ea4e042-0716-42e3-a513-92d6917fb51c"). InnerVolumeSpecName "kube-api-access-drb25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:58.116969 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.116943 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb" pod "7fb26790-a705-4a6b-88ca-96804ad613ba" (UID: "7fb26790-a705-4a6b-88ca-96804ad613ba"). InnerVolumeSpecName "kube-api-access-gfnxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:21:58.118702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.118530 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drb25\" (UniqueName: \"kubernetes.io/projected/7ea4e042-0716-42e3-a513-92d6917fb51c-kube-api-access-drb25\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:58.118702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.118561 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnxb\" (UniqueName: \"kubernetes.io/projected/7fb26790-a705-4a6b-88ca-96804ad613ba-kube-api-access-gfnxb\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:58.118702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.118575 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea4e042-0716-42e3-a513-92d6917fb51c-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:58.118702 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.118590 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb26790-a705-4a6b-88ca-96804ad613ba-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:21:58.351811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.351769 5951 generic.go:356] "Generic (PLEG): container finished" podID="7fb26790-a705-4a6b-88ca-96804ad613ba" containerID="5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498" exitCode=0 Nov 03 08:21:58.351811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.351807 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" Nov 03 08:21:58.352017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.351842 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" event={"ID":"7fb26790-a705-4a6b-88ca-96804ad613ba","Type":"ContainerDied","Data":"5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498"} Nov 03 08:21:58.352017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.351896 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r" event={"ID":"7fb26790-a705-4a6b-88ca-96804ad613ba","Type":"ContainerDied","Data":"94d796009e512aec6c23b9c2f74546991fc24b2683b9e000d8851fe77c7b1b51"} Nov 03 08:21:58.352017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.351919 5951 scope.go:122] "RemoveContainer" containerID="5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498" Nov 03 08:21:58.353860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.353834 5951 generic.go:356] "Generic (PLEG): container finished" podID="7ea4e042-0716-42e3-a513-92d6917fb51c" containerID="e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9" exitCode=0 Nov 03 08:21:58.353981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.353927 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" Nov 03 08:21:58.354115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.353924 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" event={"ID":"7ea4e042-0716-42e3-a513-92d6917fb51c","Type":"ContainerDied","Data":"e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9"} Nov 03 08:21:58.354193 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.354154 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn" event={"ID":"7ea4e042-0716-42e3-a513-92d6917fb51c","Type":"ContainerDied","Data":"244d309eb70d2c07fe4160818530493c184db7e019fa46db51f9179a6f17206c"} Nov 03 08:21:58.373269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.373241 5951 scope.go:122] "RemoveContainer" containerID="5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498" Nov 03 08:21:58.373628 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:58.373603 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498\": container with ID starting with 5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498 not found: ID does not exist" containerID="5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498" Nov 03 08:21:58.373800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.373635 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498"} err="failed to get container status \"5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498\": rpc error: code = NotFound desc = could not find container \"5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498\": container with ID starting with 5daad69c13c6a6cef76521115f6116ed543dac70c02119ec563e6757cf84f498 not found: ID does not exist" Nov 03 08:21:58.373800 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.373701 5951 scope.go:122] "RemoveContainer" containerID="e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9" Nov 03 08:21:58.380107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.380070 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r"] Nov 03 08:21:58.386097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.386074 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c8a52-65f44f8fc9-2vw2r"] Nov 03 08:21:58.388519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.388499 5951 scope.go:122] "RemoveContainer" containerID="e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9" Nov 03 08:21:58.388873 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:21:58.388853 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9\": container with ID starting with e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9 not found: ID does not exist" containerID="e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9" Nov 03 08:21:58.388952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.388879 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9"} err="failed to get container status \"e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9\": rpc error: code = NotFound desc = could not find container \"e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9\": container with ID starting with e029aac7ac3b932b75f93f8de9038f5a326bb6439e4d4fe618ebc80a992928d9 not found: ID does not exist" Nov 03 08:21:58.398569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.398543 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn"] Nov 03 08:21:58.406174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.406153 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c8a52-58c4975b87-56bbn"] Nov 03 08:21:58.745199 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.745115 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea4e042-0716-42e3-a513-92d6917fb51c" path="/var/lib/kubelet/pods/7ea4e042-0716-42e3-a513-92d6917fb51c/volumes" Nov 03 08:21:58.745823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:21:58.745803 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb26790-a705-4a6b-88ca-96804ad613ba" path="/var/lib/kubelet/pods/7fb26790-a705-4a6b-88ca-96804ad613ba/volumes" Nov 03 08:22:02.281963 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:02.281925 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:22:02.814062 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:02.814024 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv"] Nov 03 08:22:02.814337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:02.814309 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" containerID="cri-o://0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4" gracePeriod=30 Nov 03 08:22:08.090716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.090632 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:22:08.127623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.127590 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.127828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.127695 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.127828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.127740 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.127828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.127766 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.128003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.127836 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.128003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.127942 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.128120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128086 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.128203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128184 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.128272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128231 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token\") pod \"bc18256b-bc31-42da-a928-9e468c2ba641\" (UID: \"bc18256b-bc31-42da-a928-9e468c2ba641\") " Nov 03 08:22:08.128272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128080 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:08.129137 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128596 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:08.129137 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128696 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:08.129137 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.128859 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:08.129429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.129323 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.129429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.129342 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.129429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.129356 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.129429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.129371 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc18256b-bc31-42da-a928-9e468c2ba641-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.129429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.129430 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:08.132686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.130863 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:08.132686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.131223 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:22:08.132686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.131262 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:08.133003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.132980 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp" pod "bc18256b-bc31-42da-a928-9e468c2ba641" (UID: "bc18256b-bc31-42da-a928-9e468c2ba641"). InnerVolumeSpecName "kube-api-access-cslwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:08.237441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.237407 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc18256b-bc31-42da-a928-9e468c2ba641-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.237441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.237442 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.237763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.237456 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.237763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.237468 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc18256b-bc31-42da-a928-9e468c2ba641-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.237763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.237482 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cslwp\" (UniqueName: \"kubernetes.io/projected/bc18256b-bc31-42da-a928-9e468c2ba641-kube-api-access-cslwp\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:08.546218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.546166 5951 generic.go:356] "Generic (PLEG): container finished" podID="bc18256b-bc31-42da-a928-9e468c2ba641" containerID="0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4" exitCode=0 Nov 03 08:22:08.546440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.546233 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" Nov 03 08:22:08.546440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.546250 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" event={"ID":"bc18256b-bc31-42da-a928-9e468c2ba641","Type":"ContainerDied","Data":"0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4"} Nov 03 08:22:08.546440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.546308 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv" event={"ID":"bc18256b-bc31-42da-a928-9e468c2ba641","Type":"ContainerDied","Data":"71e29e3ba49d095b09073d93357ebaeb5b6fac78df43eb197ad688fa7801fefd"} Nov 03 08:22:08.546440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.546337 5951 scope.go:122] "RemoveContainer" containerID="0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4" Nov 03 08:22:08.560673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.560628 5951 scope.go:122] "RemoveContainer" containerID="0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4" Nov 03 08:22:08.561010 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:08.560987 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4\": container with ID starting with 0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4 not found: ID does not exist" containerID="0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4" Nov 03 08:22:08.561087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.561020 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4"} err="failed to get container status \"0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4\": rpc error: code = NotFound desc = could not find container \"0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4\": container with ID starting with 0cb5fa9314e7d068dfcef51ac455e89eeb88e7138be523e827bafa798f2cd0e4 not found: ID does not exist" Nov 03 08:22:08.620849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.620060 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv"] Nov 03 08:22:08.636718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.636671 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-b79hv"] Nov 03 08:22:08.742280 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:08.742241 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" path="/var/lib/kubelet/pods/bc18256b-bc31-42da-a928-9e468c2ba641/volumes" Nov 03 08:22:11.561587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:11.561531 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq"] Nov 03 08:22:11.569325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:11.569293 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f6f94-pr7fq"] Nov 03 08:22:12.740910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:12.740878 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ec46b2-7625-4160-9529-b0f6f780b7fb" path="/var/lib/kubelet/pods/98ec46b2-7625-4160-9529-b0f6f780b7fb/volumes" Nov 03 08:22:15.204543 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:15.204505 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl"] Nov 03 08:22:15.224462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:15.220872 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8"] Nov 03 08:22:20.205346 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:20.205279 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerName="payload-processing" containerID="cri-o://4c6b9e50e17ee737f64987604d95d70c7d9bfa992f9b40cf008167c390e8106d" gracePeriod=25 Nov 03 08:22:20.221569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:20.221519 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" podUID="67f93310-b389-418c-bde6-4867568cce3b" containerName="payload-pre-processing" containerID="cri-o://a37d9ed55ca38d585334c5bdc95b74f8550f34daa37347303cd07aa1ae70320d" gracePeriod=25 Nov 03 08:22:25.301602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:25.301560 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" podUID="67f93310-b389-418c-bde6-4867568cce3b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:22:25.302543 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:25.301603 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:22:27.181466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.177712 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn"] Nov 03 08:22:27.181466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.178104 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" podUID="169c6de9-86c4-4ccb-9204-9e37381ea007" containerName="maas-api" containerID="cri-o://0a602e3c785c94dc1027252ddd31eb8324fe13987dd6b33f9bbb9d5e8bac05ee" gracePeriod=30 Nov 03 08:22:27.856908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.856877 5951 generic.go:356] "Generic (PLEG): container finished" podID="169c6de9-86c4-4ccb-9204-9e37381ea007" containerID="0a602e3c785c94dc1027252ddd31eb8324fe13987dd6b33f9bbb9d5e8bac05ee" exitCode=0 Nov 03 08:22:27.857094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.856946 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" event={"ID":"169c6de9-86c4-4ccb-9204-9e37381ea007","Type":"ContainerDied","Data":"0a602e3c785c94dc1027252ddd31eb8324fe13987dd6b33f9bbb9d5e8bac05ee"} Nov 03 08:22:27.857094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.857017 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" event={"ID":"169c6de9-86c4-4ccb-9204-9e37381ea007","Type":"ContainerDied","Data":"7395d8c895346506e3ad769f719ca0801c65a38a89153ade21e53f21b4eb46fd"} Nov 03 08:22:27.857094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.857039 5951 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7395d8c895346506e3ad769f719ca0801c65a38a89153ade21e53f21b4eb46fd" Nov 03 08:22:27.875815 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.875780 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:22:27.996937 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.996894 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k\" (UniqueName: \"kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k\") pod \"169c6de9-86c4-4ccb-9204-9e37381ea007\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " Nov 03 08:22:27.997146 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.997021 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca\") pod \"169c6de9-86c4-4ccb-9204-9e37381ea007\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " Nov 03 08:22:27.997264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.997241 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls\") pod \"169c6de9-86c4-4ccb-9204-9e37381ea007\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " Nov 03 08:22:27.997417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.997357 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs\") pod \"169c6de9-86c4-4ccb-9204-9e37381ea007\" (UID: \"169c6de9-86c4-4ccb-9204-9e37381ea007\") " Nov 03 08:22:27.997495 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.997442 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca" pod "169c6de9-86c4-4ccb-9204-9e37381ea007" (UID: "169c6de9-86c4-4ccb-9204-9e37381ea007"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:27.998243 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:27.998213 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c6de9-86c4-4ccb-9204-9e37381ea007-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:28.000210 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.000183 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs" pod "169c6de9-86c4-4ccb-9204-9e37381ea007" (UID: "169c6de9-86c4-4ccb-9204-9e37381ea007"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:22:28.000497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.000473 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k" pod "169c6de9-86c4-4ccb-9204-9e37381ea007" (UID: "169c6de9-86c4-4ccb-9204-9e37381ea007"). InnerVolumeSpecName "kube-api-access-jf76k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:28.000852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.000830 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls" pod "169c6de9-86c4-4ccb-9204-9e37381ea007" (UID: "169c6de9-86c4-4ccb-9204-9e37381ea007"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:22:28.100529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.100429 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf76k\" (UniqueName: \"kubernetes.io/projected/169c6de9-86c4-4ccb-9204-9e37381ea007-kube-api-access-jf76k\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:28.100529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.100458 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:28.100529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.100469 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169c6de9-86c4-4ccb-9204-9e37381ea007-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:28.865786 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.865750 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn" Nov 03 08:22:28.904666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.889265 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn"] Nov 03 08:22:28.904666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:28.897525 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a6481-549597697-24fbn"] Nov 03 08:22:30.459678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:30.459589 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp"] Nov 03 08:22:30.460520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:30.460483 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" podUID="f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" containerName="maas-api" containerID="cri-o://f37b1a6a9868ac205d33a16f8de2ec940a1c4b2fe2610f5cabd3f282e89da9ae" gracePeriod=30 Nov 03 08:22:31.160160 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.160131 5951 generic.go:356] "Generic (PLEG): container finished" podID="f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" containerID="f37b1a6a9868ac205d33a16f8de2ec940a1c4b2fe2610f5cabd3f282e89da9ae" exitCode=0 Nov 03 08:22:31.165345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.165310 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169c6de9-86c4-4ccb-9204-9e37381ea007" path="/var/lib/kubelet/pods/169c6de9-86c4-4ccb-9204-9e37381ea007/volumes" Nov 03 08:22:31.165838 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.165815 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" event={"ID":"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575","Type":"ContainerDied","Data":"f37b1a6a9868ac205d33a16f8de2ec940a1c4b2fe2610f5cabd3f282e89da9ae"} Nov 03 08:22:31.192403 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.192377 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:22:31.379066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.379034 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls\") pod \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " Nov 03 08:22:31.379266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.379094 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca\") pod \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " Nov 03 08:22:31.379266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.379221 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs\") pod \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " Nov 03 08:22:31.379458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.379443 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7\" (UniqueName: \"kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7\") pod \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\" (UID: \"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575\") " Nov 03 08:22:31.379576 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.379472 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca" pod "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" (UID: "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:31.380209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.380188 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:31.381462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.381439 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls" pod "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" (UID: "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:22:31.381565 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.381493 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs" pod "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" (UID: "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:22:31.381808 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.381789 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7" pod "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" (UID: "f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575"). InnerVolumeSpecName "kube-api-access-sjjn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:31.481758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.481673 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjjn7\" (UniqueName: \"kubernetes.io/projected/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-kube-api-access-sjjn7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:31.481758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.481704 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:31.481758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:31.481715 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:31.957945 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:31.957898 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:22:32.172538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:32.172513 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" Nov 03 08:22:32.172744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:32.172513 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp" event={"ID":"f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575","Type":"ContainerDied","Data":"e37bc4695714217dfbc11935058a1bbea982328b7274a6185af38572cf564db4"} Nov 03 08:22:32.172744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:32.172638 5951 scope.go:122] "RemoveContainer" containerID="f37b1a6a9868ac205d33a16f8de2ec940a1c4b2fe2610f5cabd3f282e89da9ae" Nov 03 08:22:32.198891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:32.198867 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp"] Nov 03 08:22:32.206406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:32.206376 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ba69a0-7b97b5457c-qvjhp"] Nov 03 08:22:32.743240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:32.743203 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" path="/var/lib/kubelet/pods/f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575/volumes" Nov 03 08:22:34.761956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:34.761923 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6"] Nov 03 08:22:34.762357 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:34.762138 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" containerID="cri-o://f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67" gracePeriod=30 Nov 03 08:22:35.311063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:35.310619 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:22:35.311717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:35.311627 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" podUID="67f93310-b389-418c-bde6-4867568cce3b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:22:38.927109 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:38.927072 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d"] Nov 03 08:22:38.927615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:38.927339 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" containerID="cri-o://7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10" gracePeriod=30 Nov 03 08:22:38.932875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:38.932837 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn"] Nov 03 08:22:38.933042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:38.933021 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" containerID="cri-o://71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a" gracePeriod=30 Nov 03 08:22:40.150596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.150562 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:22:40.262695 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.262661 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263006 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.262989 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263108 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:40.263287 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263172 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263287 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263227 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263296 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263380 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263548 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263446 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:40.263610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263552 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263622 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.263724 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263636 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:40.263863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.263721 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket\") pod \"ec7c1dda-db35-485e-8a09-a43d79f743d4\" (UID: \"ec7c1dda-db35-485e-8a09-a43d79f743d4\") " Nov 03 08:22:40.264256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.264233 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:40.264610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.264585 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:40.265385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.264743 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.265385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.264771 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.265385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.264787 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7c1dda-db35-485e-8a09-a43d79f743d4-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.265385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.264801 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.271889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.271837 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:22:40.274444 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.271905 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:40.274444 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.272025 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:40.275371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.275302 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v" pod "ec7c1dda-db35-485e-8a09-a43d79f743d4" (UID: "ec7c1dda-db35-485e-8a09-a43d79f743d4"). InnerVolumeSpecName "kube-api-access-8z74v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:40.366441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.366334 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.366441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.366380 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.366441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.366392 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7c1dda-db35-485e-8a09-a43d79f743d4-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.366441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.366402 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z74v\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-kube-api-access-8z74v\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.366441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.366419 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7c1dda-db35-485e-8a09-a43d79f743d4-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.379739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.378925 5951 generic.go:356] "Generic (PLEG): container finished" podID="67f93310-b389-418c-bde6-4867568cce3b" containerID="a37d9ed55ca38d585334c5bdc95b74f8550f34daa37347303cd07aa1ae70320d" exitCode=0 Nov 03 08:22:40.379739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.379011 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" event={"ID":"67f93310-b389-418c-bde6-4867568cce3b","Type":"ContainerDied","Data":"a37d9ed55ca38d585334c5bdc95b74f8550f34daa37347303cd07aa1ae70320d"} Nov 03 08:22:40.383410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.383374 5951 generic.go:356] "Generic (PLEG): container finished" podID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerID="f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67" exitCode=0 Nov 03 08:22:40.383716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.383630 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" event={"ID":"ec7c1dda-db35-485e-8a09-a43d79f743d4","Type":"ContainerDied","Data":"f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67"} Nov 03 08:22:40.383716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.383710 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" event={"ID":"ec7c1dda-db35-485e-8a09-a43d79f743d4","Type":"ContainerDied","Data":"a14be07cc37069a13a5b2e6b41fb42a28f18c4186abcaeb3181c5720edea7b0a"} Nov 03 08:22:40.383850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.383744 5951 scope.go:122] "RemoveContainer" containerID="f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67" Nov 03 08:22:40.383949 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.383933 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6" Nov 03 08:22:40.410562 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.410530 5951 generic.go:356] "Generic (PLEG): container finished" podID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerID="4c6b9e50e17ee737f64987604d95d70c7d9bfa992f9b40cf008167c390e8106d" exitCode=0 Nov 03 08:22:40.411019 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.410979 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" event={"ID":"fae3a0c9-d75e-4b8a-9552-e49ea13d9737","Type":"ContainerDied","Data":"4c6b9e50e17ee737f64987604d95d70c7d9bfa992f9b40cf008167c390e8106d"} Nov 03 08:22:40.736923 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.736878 5951 scope.go:122] "RemoveContainer" containerID="7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725" Nov 03 08:22:40.738549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.738522 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6"] Nov 03 08:22:40.758108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.758044 5951 scope.go:122] "RemoveContainer" containerID="f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67" Nov 03 08:22:40.759101 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:40.759011 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67\": container with ID starting with f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67 not found: ID does not exist" containerID="f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67" Nov 03 08:22:40.759101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.759054 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67"} err="failed to get container status \"f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67\": rpc error: code = NotFound desc = could not find container \"f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67\": container with ID starting with f807057a79a18e82ba57de0804f7fe2b3c79f72debabf028856b0880e104fc67 not found: ID does not exist" Nov 03 08:22:40.759101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.759077 5951 scope.go:122] "RemoveContainer" containerID="7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725" Nov 03 08:22:40.759363 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:40.759334 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725\": container with ID starting with 7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725 not found: ID does not exist" containerID="7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725" Nov 03 08:22:40.759441 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.759366 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725"} err="failed to get container status \"7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725\": rpc error: code = NotFound desc = could not find container \"7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725\": container with ID starting with 7b9f3b1512504670294165cb9950458343c1aca653160a6757fe5d5bc8080725 not found: ID does not exist" Nov 03 08:22:40.778371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.778272 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a6481-openshift-default-b767898cc-fx5m6"] Nov 03 08:22:40.819802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.818842 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:22:40.933633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.933158 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:22:40.972359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.972327 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8\" (UniqueName: \"kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8\") pod \"67f93310-b389-418c-bde6-4867568cce3b\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " Nov 03 08:22:40.972543 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.972382 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume\") pod \"67f93310-b389-418c-bde6-4867568cce3b\" (UID: \"67f93310-b389-418c-bde6-4867568cce3b\") " Nov 03 08:22:40.972850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.972826 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume" pod "67f93310-b389-418c-bde6-4867568cce3b" (UID: "67f93310-b389-418c-bde6-4867568cce3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:40.973286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.973266 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f93310-b389-418c-bde6-4867568cce3b-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:40.975050 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:40.975025 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8" pod "67f93310-b389-418c-bde6-4867568cce3b" (UID: "67f93310-b389-418c-bde6-4867568cce3b"). InnerVolumeSpecName "kube-api-access-d5mf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:41.074463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.074428 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume\") pod \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " Nov 03 08:22:41.074685 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.074625 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l\" (UniqueName: \"kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l\") pod \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\" (UID: \"fae3a0c9-d75e-4b8a-9552-e49ea13d9737\") " Nov 03 08:22:41.074892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.074867 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume" pod "fae3a0c9-d75e-4b8a-9552-e49ea13d9737" (UID: "fae3a0c9-d75e-4b8a-9552-e49ea13d9737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:41.075468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.075451 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5mf8\" (UniqueName: \"kubernetes.io/projected/67f93310-b389-418c-bde6-4867568cce3b-kube-api-access-d5mf8\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:41.075520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.075471 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:41.076892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.076868 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l" pod "fae3a0c9-d75e-4b8a-9552-e49ea13d9737" (UID: "fae3a0c9-d75e-4b8a-9552-e49ea13d9737"). InnerVolumeSpecName "kube-api-access-jtq6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:41.177756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.177718 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtq6l\" (UniqueName: \"kubernetes.io/projected/fae3a0c9-d75e-4b8a-9552-e49ea13d9737-kube-api-access-jtq6l\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:41.424768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.424735 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" Nov 03 08:22:41.424768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.424743 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8" event={"ID":"67f93310-b389-418c-bde6-4867568cce3b","Type":"ContainerDied","Data":"d253fe6bcf1f4ef8eb56f9daaaef50427504911f1977961a13f3c147f1ac89aa"} Nov 03 08:22:41.425028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.424817 5951 scope.go:122] "RemoveContainer" containerID="a37d9ed55ca38d585334c5bdc95b74f8550f34daa37347303cd07aa1ae70320d" Nov 03 08:22:41.428939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.428866 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" Nov 03 08:22:41.428939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.428885 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl" event={"ID":"fae3a0c9-d75e-4b8a-9552-e49ea13d9737","Type":"ContainerDied","Data":"9a2093b61a6c9232569f61848a204aad5ff9a49abdb8d7b54f4da663334be52a"} Nov 03 08:22:41.448622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.448594 5951 scope.go:122] "RemoveContainer" containerID="4c6b9e50e17ee737f64987604d95d70c7d9bfa992f9b40cf008167c390e8106d" Nov 03 08:22:41.454270 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.454244 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8"] Nov 03 08:22:41.462985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.462952 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ba69a0-6f646bfd99-bnsb8"] Nov 03 08:22:41.510060 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.510023 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl"] Nov 03 08:22:41.514582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:41.514554 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ba69a0-6dd9685599-mcxhl"] Nov 03 08:22:42.824669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:42.821698 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f93310-b389-418c-bde6-4867568cce3b" path="/var/lib/kubelet/pods/67f93310-b389-418c-bde6-4867568cce3b/volumes" Nov 03 08:22:42.824669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:42.823023 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" path="/var/lib/kubelet/pods/ec7c1dda-db35-485e-8a09-a43d79f743d4/volumes" Nov 03 08:22:42.824669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:42.823789 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" path="/var/lib/kubelet/pods/fae3a0c9-d75e-4b8a-9552-e49ea13d9737/volumes" Nov 03 08:22:42.971668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:42.967742 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:22:43.025770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:43.025728 5951 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:22:43.161817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:43.161717 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:22:43.225785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:43.225737 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:22:43.261223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:43.260594 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:22:43.878141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:43.878104 5951 patch_prober.go:29] interesting pod/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:22:43.893680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:43.887744 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:22:44.379917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.368906 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.445615 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.445859 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.445897 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.445953 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.446237 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.446296 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.446346 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.446376 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.448235 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.446473 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h\") pod \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\" (UID: \"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5\") " Nov 03 08:22:44.458215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.456343 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.458215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.456698 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.458215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.456952 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.458215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.457224 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:44.458215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.458154 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.461181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.461156 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:44.462305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.462281 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:22:44.463591 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.463560 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.466954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.465607 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h" pod "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" (UID: "dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5"). InnerVolumeSpecName "kube-api-access-27x8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:44.486299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.486274 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:22:44.548886 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548854 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548915 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548931 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548943 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548956 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548969 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27x8h\" (UniqueName: \"kubernetes.io/projected/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-kube-api-access-27x8h\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548983 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.548995 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.549077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.549009 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.574690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.574658 5951 generic.go:356] "Generic (PLEG): container finished" podID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerID="7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10" exitCode=0 Nov 03 08:22:44.574881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.574710 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" Nov 03 08:22:44.574881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.574739 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" event={"ID":"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5","Type":"ContainerDied","Data":"7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10"} Nov 03 08:22:44.574881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.574805 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d" event={"ID":"dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5","Type":"ContainerDied","Data":"9fb525d8ac99c2b46b75254fcbbb1ddb897acb1bb72dad52749dbb426d23c270"} Nov 03 08:22:44.574881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.574837 5951 scope.go:122] "RemoveContainer" containerID="7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10" Nov 03 08:22:44.579419 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.579394 5951 generic.go:356] "Generic (PLEG): container finished" podID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerID="71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a" exitCode=0 Nov 03 08:22:44.579563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.579515 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" event={"ID":"976598fb-dba9-4fb3-a581-7c780f7c56cc","Type":"ContainerDied","Data":"71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a"} Nov 03 08:22:44.579563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.579548 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" event={"ID":"976598fb-dba9-4fb3-a581-7c780f7c56cc","Type":"ContainerDied","Data":"c3fc4b66b87521da5fbc21b1b7051d2e5fe04ca4aea64ef84a921ad55c37d0eb"} Nov 03 08:22:44.579743 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.579610 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn" Nov 03 08:22:44.648839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.648811 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d"] Nov 03 08:22:44.649563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.649536 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.649780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.649763 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.649954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.649940 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.650069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.650057 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.650242 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.650228 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.650504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.650488 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.650733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.650717 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.650926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.650910 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.651123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.651107 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket\") pod \"976598fb-dba9-4fb3-a581-7c780f7c56cc\" (UID: \"976598fb-dba9-4fb3-a581-7c780f7c56cc\") " Nov 03 08:22:44.652319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.652297 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.652620 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.652605 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.653063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.653046 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:22:44.654447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.654417 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.654592 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.654567 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.654809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.654795 5951 scope.go:122] "RemoveContainer" containerID="5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62" Nov 03 08:22:44.658732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.658680 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-ztb8d"] Nov 03 08:22:44.666064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.665809 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:22:44.666064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.665839 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:44.666899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.666818 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "kube-api-access-5b27z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:22:44.667845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.667810 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo" pod "976598fb-dba9-4fb3-a581-7c780f7c56cc" (UID: "976598fb-dba9-4fb3-a581-7c780f7c56cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:22:44.675847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.675613 5951 scope.go:122] "RemoveContainer" containerID="7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10" Nov 03 08:22:44.675952 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:44.675900 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10\": container with ID starting with 7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10 not found: ID does not exist" containerID="7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10" Nov 03 08:22:44.676012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.675952 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10"} err="failed to get container status \"7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10\": rpc error: code = NotFound desc = could not find container \"7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10\": container with ID starting with 7490f0cff6e179908df4bab3ad862c0ec2e186dd90e5722c99765759f6b64b10 not found: ID does not exist" Nov 03 08:22:44.676012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.675968 5951 scope.go:122] "RemoveContainer" containerID="5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62" Nov 03 08:22:44.676356 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:44.676208 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62\": container with ID starting with 5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62 not found: ID does not exist" containerID="5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62" Nov 03 08:22:44.676356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.676242 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62"} err="failed to get container status \"5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62\": rpc error: code = NotFound desc = could not find container \"5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62\": container with ID starting with 5ec9b8211cd95431141dc6770b25ff35f4cb0c80a47c08b21a30e1b81a5d5c62 not found: ID does not exist" Nov 03 08:22:44.676356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.676262 5951 scope.go:122] "RemoveContainer" containerID="71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a" Nov 03 08:22:44.688667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.688637 5951 scope.go:122] "RemoveContainer" containerID="71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a" Nov 03 08:22:44.688970 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:22:44.688940 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a\": container with ID starting with 71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a not found: ID does not exist" containerID="71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a" Nov 03 08:22:44.689076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.688977 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a"} err="failed to get container status \"71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a\": rpc error: code = NotFound desc = could not find container \"71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a\": container with ID starting with 71519dbc52d88352777d4adb1ed39bcce632c869aff419835ff986eb64112b6a not found: ID does not exist" Nov 03 08:22:44.744582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.744537 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" path="/var/lib/kubelet/pods/dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5/volumes" Nov 03 08:22:44.754739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754713 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754738 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754748 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754756 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754767 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754775 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/976598fb-dba9-4fb3-a581-7c780f7c56cc-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754784 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754792 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b27z\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-kube-api-access-5b27z\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.754907 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.754801 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/976598fb-dba9-4fb3-a581-7c780f7c56cc-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:22:44.934219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.934179 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn"] Nov 03 08:22:44.938254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:44.938219 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ba69a0-openshift-default-6d5c8696c7-9wmhn"] Nov 03 08:22:46.742796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:46.742765 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" path="/var/lib/kubelet/pods/976598fb-dba9-4fb3-a581-7c780f7c56cc/volumes" Nov 03 08:22:49.583406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:49.583373 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v"] Nov 03 08:22:54.584516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.584438 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" containerName="payload-processing" containerID="cri-o://6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a" gracePeriod=25 Nov 03 08:22:54.795015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.794984 5951 generic.go:356] "Generic (PLEG): container finished" podID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerID="c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f" exitCode=0 Nov 03 08:22:54.795209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.795054 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" event={"ID":"11d81aa6-a90e-4457-8dd9-41a95c107643","Type":"ContainerDied","Data":"c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f"} Nov 03 08:22:54.795634 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.795612 5951 scope.go:122] "RemoveContainer" containerID="c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f" Nov 03 08:22:54.797868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.797827 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:22:54.797992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.797923 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:22:54.797992 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:54.797975 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:22:55.827311 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:55.827264 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" event={"ID":"11d81aa6-a90e-4457-8dd9-41a95c107643","Type":"ContainerStarted","Data":"b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648"} Nov 03 08:22:56.179029 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:56.178931 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:22:56.228725 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:56.225781 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:22:56.229516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:56.229483 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:22:56.229976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:56.229943 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:22:57.239681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:57.231038 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:22:57.239681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:57.231098 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:22:58.231734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:58.228821 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:22:58.231734 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:58.228875 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:22:59.225043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:59.224965 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:22:59.225043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:22:59.225023 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:23:00.227180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:00.227141 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:23:00.227828 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:00.227793 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:23:01.234673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:01.231236 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 08:23:01.234673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:01.231376 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 08:23:01.585856 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:01.581400 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:23:01.966508 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:01.966465 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:23:02.231215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:02.231131 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:23:02.232367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:02.232340 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:23:05.277732 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.277695 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w"] Nov 03 08:23:05.279798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.279770 5951 cpu_manager.go:414] "containerMap: removing container" podUID="7fb26790-a705-4a6b-88ca-96804ad613ba" containerName="payload-pre-processing" Nov 03 08:23:05.279965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.279951 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb26790-a705-4a6b-88ca-96804ad613ba" containerName="payload-pre-processing" Nov 03 08:23:05.280059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.280049 5951 cpu_manager.go:414] "containerMap: removing container" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" Nov 03 08:23:05.280143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.280132 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" Nov 03 08:23:05.280233 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.280223 5951 cpu_manager.go:414] "containerMap: removing container" podUID="7ea4e042-0716-42e3-a513-92d6917fb51c" containerName="payload-processing" Nov 03 08:23:05.280315 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.280301 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea4e042-0716-42e3-a513-92d6917fb51c" containerName="payload-processing" Nov 03 08:23:05.280436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.280424 5951 cpu_manager.go:414] "containerMap: removing container" podUID="169c6de9-86c4-4ccb-9204-9e37381ea007" containerName="maas-api" Nov 03 08:23:05.284684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.280513 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169c6de9-86c4-4ccb-9204-9e37381ea007" containerName="maas-api" Nov 03 08:23:05.284874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.284853 5951 cpu_manager.go:414] "containerMap: removing container" podUID="67f93310-b389-418c-bde6-4867568cce3b" containerName="payload-pre-processing" Nov 03 08:23:05.284966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.284954 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f93310-b389-418c-bde6-4867568cce3b" containerName="payload-pre-processing" Nov 03 08:23:05.285052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285042 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" Nov 03 08:23:05.285131 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285121 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" Nov 03 08:23:05.285233 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285222 5951 cpu_manager.go:414] "containerMap: removing container" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" Nov 03 08:23:05.285311 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285301 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" Nov 03 08:23:05.285416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285406 5951 cpu_manager.go:414] "containerMap: removing container" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" Nov 03 08:23:05.285494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285484 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" Nov 03 08:23:05.285603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285594 5951 cpu_manager.go:414] "containerMap: removing container" podUID="96897ed3-96a4-46a1-87aa-99333dae25eb" containerName="revoke-keys" Nov 03 08:23:05.285717 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285704 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96897ed3-96a4-46a1-87aa-99333dae25eb" containerName="revoke-keys" Nov 03 08:23:05.285823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285812 5951 cpu_manager.go:414] "containerMap: removing container" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" Nov 03 08:23:05.285893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285883 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" Nov 03 08:23:05.285980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.285971 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" Nov 03 08:23:05.292568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.292547 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" Nov 03 08:23:05.292770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.292755 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerName="payload-processing" Nov 03 08:23:05.292889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.292877 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerName="payload-processing" Nov 03 08:23:05.292981 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.292971 5951 cpu_manager.go:414] "containerMap: removing container" podUID="98ec46b2-7625-4160-9529-b0f6f780b7fb" containerName="revoke-keys" Nov 03 08:23:05.293061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.293052 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ec46b2-7625-4160-9529-b0f6f780b7fb" containerName="revoke-keys" Nov 03 08:23:05.293149 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.293139 5951 cpu_manager.go:414] "containerMap: removing container" podUID="f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" containerName="maas-api" Nov 03 08:23:05.293221 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.293211 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" containerName="maas-api" Nov 03 08:23:05.293859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.293837 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae3a0c9-d75e-4b8a-9552-e49ea13d9737" containerName="payload-processing" Nov 03 08:23:05.293982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.293970 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ec46b2-7625-4160-9529-b0f6f780b7fb" containerName="revoke-keys" Nov 03 08:23:05.294072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.294063 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a9e81b-9bc9-4de9-8e7c-98ecf34b8575" containerName="maas-api" Nov 03 08:23:05.294152 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.294143 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb26790-a705-4a6b-88ca-96804ad613ba" containerName="payload-pre-processing" Nov 03 08:23:05.294226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.294217 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc18256b-bc31-42da-a928-9e468c2ba641" containerName="istio-proxy" Nov 03 08:23:05.294297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.294289 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea4e042-0716-42e3-a513-92d6917fb51c" containerName="payload-processing" Nov 03 08:23:05.294377 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.294368 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="169c6de9-86c4-4ccb-9204-9e37381ea007" containerName="maas-api" Nov 03 08:23:05.298683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.298666 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f93310-b389-418c-bde6-4867568cce3b" containerName="payload-pre-processing" Nov 03 08:23:05.298822 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.298811 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" Nov 03 08:23:05.298902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.298894 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" Nov 03 08:23:05.298973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.298965 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5a1c18-2ec4-455a-8e3f-b1d27856c8a5" containerName="istio-proxy" Nov 03 08:23:05.299044 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.299037 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="96897ed3-96a4-46a1-87aa-99333dae25eb" containerName="revoke-keys" Nov 03 08:23:05.299105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.299098 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="976598fb-dba9-4fb3-a581-7c780f7c56cc" containerName="istio-proxy" Nov 03 08:23:05.315291 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.315262 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w"] Nov 03 08:23:05.335690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.315916 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.335690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.333585 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-109070-openshift-default-dockercfg-5wqds\"" Nov 03 08:23:05.423054 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.422879 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.429459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.429428 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.429750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.429727 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.430104 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.430062 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.431277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.430275 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qph5t\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.431513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.431479 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.431707 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.431684 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.431954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.431932 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.432143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.432128 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.534739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534358 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.534739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534459 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qph5t\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.534739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534510 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.534739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534569 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.534739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534622 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.534739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534720 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.535295 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534807 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.535295 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534892 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.535295 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.534942 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.535295 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.535175 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.545192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.537042 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.545192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.538247 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.545192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.538403 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.545192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.538463 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.546498 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.546443 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.549206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.549169 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qph5t\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.550093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.550005 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.550621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.550592 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.683851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:05.683803 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:05.997487 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:23:05.997452 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901fa1d8_7cb9_4d66_8a68_36970e80813a.slice/crio-ff2778ca89f62edd478a3daeb8caed341048fcf18ca90e50c97aee5c1ee375a3 WatchSource:0}: Error finding container ff2778ca89f62edd478a3daeb8caed341048fcf18ca90e50c97aee5c1ee375a3: Status 404 returned error can't find the container with id ff2778ca89f62edd478a3daeb8caed341048fcf18ca90e50c97aee5c1ee375a3 Nov 03 08:23:06.001212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.001169 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:23:06.001331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.001270 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:23:06.001331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.001311 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:23:06.006222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.006196 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w"] Nov 03 08:23:06.959385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.959348 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" event={"ID":"901fa1d8-7cb9-4d66-8a68-36970e80813a","Type":"ContainerStarted","Data":"cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5"} Nov 03 08:23:06.959805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.959405 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" event={"ID":"901fa1d8-7cb9-4d66-8a68-36970e80813a","Type":"ContainerStarted","Data":"ff2778ca89f62edd478a3daeb8caed341048fcf18ca90e50c97aee5c1ee375a3"} Nov 03 08:23:06.986468 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:06.986334 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podStartSLOduration=1.9863156549999998 podStartE2EDuration="1.986315655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:06.979588655 +0000 UTC m=+1726.361039575" watchObservedRunningTime="2026-11-03 08:23:06.986315655 +0000 UTC m=+1726.367766571" Nov 03 08:23:07.684534 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:07.684503 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:07.685781 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:07.685754 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:07.685895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:07.685802 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:08.684826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:08.684793 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:08.685309 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:08.685281 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:09.684603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:09.684561 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:09.684805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:09.684612 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:10.687667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:10.687041 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:10.687667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:10.687135 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:11.546436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:11.546392 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:23:11.713668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:11.700830 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:11.713668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:11.700899 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:11.878737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:11.870627 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl"] Nov 03 08:23:11.948664 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:11.939589 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm"] Nov 03 08:23:12.695159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:12.694156 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:12.695159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:12.694213 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:13.686529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:13.684780 5951 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 03 08:23:13.686529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:13.684877 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 03 08:23:14.460391 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:14.460352 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4"] Nov 03 08:23:14.460613 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:14.460585 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" containerID="cri-o://51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667" gracePeriod=30 Nov 03 08:23:14.849183 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:14.849112 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:14.850222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:14.850139 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:14.852772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:14.852613 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:23:15.160004 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.159979 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:23:15.282068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.282035 5951 generic.go:356] "Generic (PLEG): container finished" podID="cdd8f6e3-3644-469e-b73b-db7758756182" containerID="6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a" exitCode=0 Nov 03 08:23:15.282222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.282078 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" Nov 03 08:23:15.282222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.282116 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" event={"ID":"cdd8f6e3-3644-469e-b73b-db7758756182","Type":"ContainerDied","Data":"6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a"} Nov 03 08:23:15.282222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.282166 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v" event={"ID":"cdd8f6e3-3644-469e-b73b-db7758756182","Type":"ContainerDied","Data":"794d8a00033c1b503d213cc1afcbccecce7a2b80de0fa1df3d94daefc05198bb"} Nov 03 08:23:15.282222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.282193 5951 scope.go:122] "RemoveContainer" containerID="6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a" Nov 03 08:23:15.295598 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.295569 5951 scope.go:122] "RemoveContainer" containerID="6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a" Nov 03 08:23:15.296120 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:15.296093 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a\": container with ID starting with 6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a not found: ID does not exist" containerID="6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a" Nov 03 08:23:15.296220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.296131 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a"} err="failed to get container status \"6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a\": rpc error: code = NotFound desc = could not find container \"6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a\": container with ID starting with 6cd32abf632a802409b541f5a7b89b95cb882981a0e614e4383fe1399d81315a not found: ID does not exist" Nov 03 08:23:15.338323 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.338292 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz\") pod \"cdd8f6e3-3644-469e-b73b-db7758756182\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " Nov 03 08:23:15.338506 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.338396 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume\") pod \"cdd8f6e3-3644-469e-b73b-db7758756182\" (UID: \"cdd8f6e3-3644-469e-b73b-db7758756182\") " Nov 03 08:23:15.338819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.338797 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume" pod "cdd8f6e3-3644-469e-b73b-db7758756182" (UID: "cdd8f6e3-3644-469e-b73b-db7758756182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:15.339055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.339013 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd8f6e3-3644-469e-b73b-db7758756182-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:15.340770 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.340744 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz" pod "cdd8f6e3-3644-469e-b73b-db7758756182" (UID: "cdd8f6e3-3644-469e-b73b-db7758756182"). InnerVolumeSpecName "kube-api-access-hpftz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:15.441446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.441350 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/cdd8f6e3-3644-469e-b73b-db7758756182-kube-api-access-hpftz\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:15.610408 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.610377 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v"] Nov 03 08:23:15.615067 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:15.615033 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-179c16-c957df445-9tz5v"] Nov 03 08:23:16.739782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:16.739746 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" path="/var/lib/kubelet/pods/cdd8f6e3-3644-469e-b73b-db7758756182/volumes" Nov 03 08:23:16.875747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:16.875699 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" podUID="6b3d384d-4065-42f2-9334-42ca4377f1da" containerName="payload-processing" containerID="cri-o://94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238" gracePeriod=25 Nov 03 08:23:16.940970 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:16.940930 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" podUID="693dc940-8e5b-4d53-94d5-06b60fca8bbe" containerName="payload-pre-processing" containerID="cri-o://958afdbc74a8eb0f1490b18430270ca2bd7e2fd07fdf51232131e5ca9fb4ec7c" gracePeriod=25 Nov 03 08:23:17.319747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.319234 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:23:17.322302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.321453 5951 generic.go:356] "Generic (PLEG): container finished" podID="6b3d384d-4065-42f2-9334-42ca4377f1da" containerID="94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238" exitCode=0 Nov 03 08:23:17.322302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.321595 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" event={"ID":"6b3d384d-4065-42f2-9334-42ca4377f1da","Type":"ContainerDied","Data":"94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238"} Nov 03 08:23:17.322302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.321689 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" event={"ID":"6b3d384d-4065-42f2-9334-42ca4377f1da","Type":"ContainerDied","Data":"01f80c31fbd90ce3913c844da061668aec285b725c4f159c59205a809a1bbbd6"} Nov 03 08:23:17.322302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.321726 5951 scope.go:122] "RemoveContainer" containerID="94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238" Nov 03 08:23:17.324868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.324829 5951 generic.go:356] "Generic (PLEG): container finished" podID="693dc940-8e5b-4d53-94d5-06b60fca8bbe" containerID="958afdbc74a8eb0f1490b18430270ca2bd7e2fd07fdf51232131e5ca9fb4ec7c" exitCode=0 Nov 03 08:23:17.325252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.325230 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" event={"ID":"693dc940-8e5b-4d53-94d5-06b60fca8bbe","Type":"ContainerDied","Data":"958afdbc74a8eb0f1490b18430270ca2bd7e2fd07fdf51232131e5ca9fb4ec7c"} Nov 03 08:23:17.347744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.346179 5951 scope.go:122] "RemoveContainer" containerID="94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238" Nov 03 08:23:17.347744 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:17.347322 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238\": container with ID starting with 94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238 not found: ID does not exist" containerID="94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238" Nov 03 08:23:17.347744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.347361 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238"} err="failed to get container status \"94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238\": rpc error: code = NotFound desc = could not find container \"94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238\": container with ID starting with 94bffb12fed562887489c93bfa7dd48a355b4d05a014bf469eef176251928238 not found: ID does not exist" Nov 03 08:23:17.462829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.462785 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7\" (UniqueName: \"kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7\") pod \"6b3d384d-4065-42f2-9334-42ca4377f1da\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " Nov 03 08:23:17.463210 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.463191 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume\") pod \"6b3d384d-4065-42f2-9334-42ca4377f1da\" (UID: \"6b3d384d-4065-42f2-9334-42ca4377f1da\") " Nov 03 08:23:17.465938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.465758 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume" pod "6b3d384d-4065-42f2-9334-42ca4377f1da" (UID: "6b3d384d-4065-42f2-9334-42ca4377f1da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:17.473091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.473057 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7" pod "6b3d384d-4065-42f2-9334-42ca4377f1da" (UID: "6b3d384d-4065-42f2-9334-42ca4377f1da"). InnerVolumeSpecName "kube-api-access-ccxn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:17.481497 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.481173 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:23:17.565706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.565630 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume\") pod \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " Nov 03 08:23:17.565929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.565906 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn\" (UniqueName: \"kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn\") pod \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\" (UID: \"693dc940-8e5b-4d53-94d5-06b60fca8bbe\") " Nov 03 08:23:17.566248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.566222 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume" pod "693dc940-8e5b-4d53-94d5-06b60fca8bbe" (UID: "693dc940-8e5b-4d53-94d5-06b60fca8bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:17.566851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.566825 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d384d-4065-42f2-9334-42ca4377f1da-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:17.566851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.566855 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693dc940-8e5b-4d53-94d5-06b60fca8bbe-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:17.567038 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.566870 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxn7\" (UniqueName: \"kubernetes.io/projected/6b3d384d-4065-42f2-9334-42ca4377f1da-kube-api-access-ccxn7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:17.568255 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.568232 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn" pod "693dc940-8e5b-4d53-94d5-06b60fca8bbe" (UID: "693dc940-8e5b-4d53-94d5-06b60fca8bbe"). InnerVolumeSpecName "kube-api-access-slldn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:17.674426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:17.672150 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slldn\" (UniqueName: \"kubernetes.io/projected/693dc940-8e5b-4d53-94d5-06b60fca8bbe-kube-api-access-slldn\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:18.038041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.038006 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2"] Nov 03 08:23:18.043144 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.043117 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2"] Nov 03 08:23:18.336398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.336309 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl" Nov 03 08:23:18.338080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.338054 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" Nov 03 08:23:18.338244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.338083 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm" event={"ID":"693dc940-8e5b-4d53-94d5-06b60fca8bbe","Type":"ContainerDied","Data":"928b5fea078c001b71ec5616b94e5ca703efdc6313218f4751971e85d48fa2c0"} Nov 03 08:23:18.338244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.338168 5951 scope.go:122] "RemoveContainer" containerID="958afdbc74a8eb0f1490b18430270ca2bd7e2fd07fdf51232131e5ca9fb4ec7c" Nov 03 08:23:18.366436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.366404 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm"] Nov 03 08:23:18.371088 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.371058 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-fe584a-85d7f944b7-lqrkm"] Nov 03 08:23:18.380866 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.380839 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl"] Nov 03 08:23:18.388863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.388836 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-fe584a-755887cfb9-6xspl"] Nov 03 08:23:18.739544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.739467 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693dc940-8e5b-4d53-94d5-06b60fca8bbe" path="/var/lib/kubelet/pods/693dc940-8e5b-4d53-94d5-06b60fca8bbe/volumes" Nov 03 08:23:18.740001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:18.739986 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3d384d-4065-42f2-9334-42ca4377f1da" path="/var/lib/kubelet/pods/6b3d384d-4065-42f2-9334-42ca4377f1da/volumes" Nov 03 08:23:19.793194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.793170 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:23:19.820661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.820617 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.820900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.820687 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.820900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.820787 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.820927 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.820961 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.821046 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.821082 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.821131 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.821202 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs\") pod \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\" (UID: \"a4853d6b-f6ba-4245-89aa-2069c46cea2a\") " Nov 03 08:23:19.821556 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.821293 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:19.822413 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.822282 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:19.822413 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.822340 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:19.822715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.822689 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:19.823108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.823088 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:19.823238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.823185 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.823400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.823249 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.823400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.823266 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.823400 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.823280 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.825560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.825150 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "kube-api-access-fzvrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:19.827001 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.826980 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:19.827120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.827106 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:23:19.827829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.827805 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy" pod "a4853d6b-f6ba-4245-89aa-2069c46cea2a" (UID: "a4853d6b-f6ba-4245-89aa-2069c46cea2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:19.924865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.924827 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.924865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.924860 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvrc\" (UniqueName: \"kubernetes.io/projected/a4853d6b-f6ba-4245-89aa-2069c46cea2a-kube-api-access-fzvrc\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.924865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.924871 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.925096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.924880 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853d6b-f6ba-4245-89aa-2069c46cea2a-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:19.925096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:19.924889 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853d6b-f6ba-4245-89aa-2069c46cea2a-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:20.408665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.397011 5951 generic.go:356] "Generic (PLEG): container finished" podID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerID="51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667" exitCode=0 Nov 03 08:23:20.408665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.397286 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" event={"ID":"a4853d6b-f6ba-4245-89aa-2069c46cea2a","Type":"ContainerDied","Data":"51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667"} Nov 03 08:23:20.408665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.397338 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" event={"ID":"a4853d6b-f6ba-4245-89aa-2069c46cea2a","Type":"ContainerDied","Data":"87251473dc7575dcfdb6fa1c66dcd157cf6452fbeb39b6dccb4bf64456f78912"} Nov 03 08:23:20.408665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.397366 5951 scope.go:122] "RemoveContainer" containerID="51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667" Nov 03 08:23:20.408665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.397484 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4" Nov 03 08:23:20.590293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.586140 5951 scope.go:122] "RemoveContainer" containerID="791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e" Nov 03 08:23:20.731858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.731824 5951 scope.go:122] "RemoveContainer" containerID="51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667" Nov 03 08:23:20.737509 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:20.737480 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667\": container with ID starting with 51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667 not found: ID does not exist" containerID="51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667" Nov 03 08:23:20.737796 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.737768 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667"} err="failed to get container status \"51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667\": rpc error: code = NotFound desc = could not find container \"51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667\": container with ID starting with 51bbaa46f34b364648311d744ad120eef5addf88fcf73c74db5c6b5679f9f667 not found: ID does not exist" Nov 03 08:23:20.738076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.738048 5951 scope.go:122] "RemoveContainer" containerID="791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e" Nov 03 08:23:20.752722 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:20.751494 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e\": container with ID starting with 791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e not found: ID does not exist" containerID="791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e" Nov 03 08:23:20.754269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.754216 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e"} err="failed to get container status \"791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e\": rpc error: code = NotFound desc = could not find container \"791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e\": container with ID starting with 791703868a8ba3094941d74a0d08c0cde8c42cb16dcea66cac5cc914c9de332e not found: ID does not exist" Nov 03 08:23:20.792721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.791542 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4"] Nov 03 08:23:20.793334 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:20.793311 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-179c16-openshift-default-5dc5b8f44c-86jp4"] Nov 03 08:23:22.742249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:22.742208 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" path="/var/lib/kubelet/pods/a4853d6b-f6ba-4245-89aa-2069c46cea2a/volumes" Nov 03 08:23:24.626569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:24.626518 5951 scope.go:122] "RemoveContainer" containerID="0a602e3c785c94dc1027252ddd31eb8324fe13987dd6b33f9bbb9d5e8bac05ee" Nov 03 08:23:25.807132 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:25.807097 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq"] Nov 03 08:23:25.812967 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:25.812938 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx"] Nov 03 08:23:25.823249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:25.823218 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7"] Nov 03 08:23:31.241666 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:31.240139 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l"] Nov 03 08:23:31.957400 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:31.957359 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:23:33.065487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.065440 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" containerID="cri-o://cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd" gracePeriod=45 Nov 03 08:23:33.066341 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.066283 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" containerID="cri-o://730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52" gracePeriod=45 Nov 03 08:23:33.202042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.201923 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 08:23:33.465837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.465810 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:23:33.504386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.504013 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:23:33.525453 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.525389 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache" pod "5472f604-f001-48f8-a95a-4680d038e507" (UID: "5472f604-f001-48f8-a95a-4680d038e507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.526075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.526050 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache\") pod \"5472f604-f001-48f8-a95a-4680d038e507\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " Nov 03 08:23:33.528155 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.528124 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm\") pod \"5472f604-f001-48f8-a95a-4680d038e507\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " Nov 03 08:23:33.528947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.528908 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir" pod "5472f604-f001-48f8-a95a-4680d038e507" (UID: "5472f604-f001-48f8-a95a-4680d038e507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.529126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.529109 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir\") pod \"5472f604-f001-48f8-a95a-4680d038e507\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " Nov 03 08:23:33.529831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.529813 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs\" (UniqueName: \"kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs\") pod \"5472f604-f001-48f8-a95a-4680d038e507\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " Nov 03 08:23:33.530020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.530003 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home\") pod \"5472f604-f001-48f8-a95a-4680d038e507\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " Nov 03 08:23:33.531799 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.530832 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks\" (UniqueName: \"kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks\") pod \"5472f604-f001-48f8-a95a-4680d038e507\" (UID: \"5472f604-f001-48f8-a95a-4680d038e507\") " Nov 03 08:23:33.532463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.531082 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm" pod "5472f604-f001-48f8-a95a-4680d038e507" (UID: "5472f604-f001-48f8-a95a-4680d038e507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.532463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.531378 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home" pod "5472f604-f001-48f8-a95a-4680d038e507" (UID: "5472f604-f001-48f8-a95a-4680d038e507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.534038 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.534014 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks" pod "5472f604-f001-48f8-a95a-4680d038e507" (UID: "5472f604-f001-48f8-a95a-4680d038e507"). InnerVolumeSpecName "kube-api-access-qg2ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:33.535187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.535170 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.535319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.535309 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.535414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.535398 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg2ks\" (UniqueName: \"kubernetes.io/projected/5472f604-f001-48f8-a95a-4680d038e507-kube-api-access-qg2ks\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.535503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.535491 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.535601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.535578 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5472f604-f001-48f8-a95a-4680d038e507-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.544094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.544064 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs" pod "5472f604-f001-48f8-a95a-4680d038e507" (UID: "5472f604-f001-48f8-a95a-4680d038e507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:33.581733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.581697 5951 generic.go:356] "Generic (PLEG): container finished" podID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerID="cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd" exitCode=0 Nov 03 08:23:33.581887 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.581873 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" Nov 03 08:23:33.582012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.581871 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" event={"ID":"9a561f60-a65e-4eeb-b7b2-b832818dc2bc","Type":"ContainerDied","Data":"cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd"} Nov 03 08:23:33.582087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.582040 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2" event={"ID":"9a561f60-a65e-4eeb-b7b2-b832818dc2bc","Type":"ContainerDied","Data":"f334db3ca21b27664edacd40bb8684fe37d47d1be30ab84017d4c2c010e05b2e"} Nov 03 08:23:33.582087 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.582067 5951 scope.go:122] "RemoveContainer" containerID="cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd" Nov 03 08:23:33.584354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.584321 5951 generic.go:356] "Generic (PLEG): container finished" podID="5472f604-f001-48f8-a95a-4680d038e507" containerID="730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52" exitCode=0 Nov 03 08:23:33.584463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.584399 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" event={"ID":"5472f604-f001-48f8-a95a-4680d038e507","Type":"ContainerDied","Data":"730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52"} Nov 03 08:23:33.584463 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.584431 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" event={"ID":"5472f604-f001-48f8-a95a-4680d038e507","Type":"ContainerDied","Data":"04c3137f7d06c2464da2494c33184f2c7313e7e70a5b296fc90f07b0104359c6"} Nov 03 08:23:33.584542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.584486 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2" Nov 03 08:23:33.609975 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.609944 5951 scope.go:122] "RemoveContainer" containerID="cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd" Nov 03 08:23:33.610878 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:33.610840 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd\": container with ID starting with cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd not found: ID does not exist" containerID="cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd" Nov 03 08:23:33.611047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.611020 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd"} err="failed to get container status \"cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd\": rpc error: code = NotFound desc = could not find container \"cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd\": container with ID starting with cc89d1080a73b41f31bb4c41e1b93a1fe616f12374ee66a0d8e706d3e6e6c9cd not found: ID does not exist" Nov 03 08:23:33.611185 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.611172 5951 scope.go:122] "RemoveContainer" containerID="730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52" Nov 03 08:23:33.624174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.624151 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2"] Nov 03 08:23:33.631737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.631703 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-w4-15a633-kserve-58b98467d-rz7r2"] Nov 03 08:23:33.635632 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.635610 5951 scope.go:122] "RemoveContainer" containerID="730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52" Nov 03 08:23:33.635988 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:33.635954 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52\": container with ID starting with 730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52 not found: ID does not exist" containerID="730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52" Nov 03 08:23:33.636136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.636000 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52"} err="failed to get container status \"730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52\": rpc error: code = NotFound desc = could not find container \"730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52\": container with ID starting with 730d2345029938d6cdf41e550d2f1b0440d74aeac6751830fe4b60923e89fd52 not found: ID does not exist" Nov 03 08:23:33.636291 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.636264 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5\" (UniqueName: \"kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5\") pod \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " Nov 03 08:23:33.636389 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.636371 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home\") pod \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " Nov 03 08:23:33.636798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.636628 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir\") pod \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " Nov 03 08:23:33.636798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.636746 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm\") pod \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " Nov 03 08:23:33.636798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.636776 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs\") pod \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " Nov 03 08:23:33.637285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.637111 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache\") pod \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\" (UID: \"9a561f60-a65e-4eeb-b7b2-b832818dc2bc\") " Nov 03 08:23:33.637897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.637457 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir" pod "9a561f60-a65e-4eeb-b7b2-b832818dc2bc" (UID: "9a561f60-a65e-4eeb-b7b2-b832818dc2bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.637897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.637689 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache" pod "9a561f60-a65e-4eeb-b7b2-b832818dc2bc" (UID: "9a561f60-a65e-4eeb-b7b2-b832818dc2bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.637897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.637874 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home" pod "9a561f60-a65e-4eeb-b7b2-b832818dc2bc" (UID: "9a561f60-a65e-4eeb-b7b2-b832818dc2bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.638097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.638040 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.638097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.638060 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5472f604-f001-48f8-a95a-4680d038e507-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.638097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.638073 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.638097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.638085 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.641845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.641372 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs" pod "9a561f60-a65e-4eeb-b7b2-b832818dc2bc" (UID: "9a561f60-a65e-4eeb-b7b2-b832818dc2bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:33.641995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.641969 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5" pod "9a561f60-a65e-4eeb-b7b2-b832818dc2bc" (UID: "9a561f60-a65e-4eeb-b7b2-b832818dc2bc"). InnerVolumeSpecName "kube-api-access-xb4q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:33.642349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.642314 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm" pod "9a561f60-a65e-4eeb-b7b2-b832818dc2bc" (UID: "9a561f60-a65e-4eeb-b7b2-b832818dc2bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:33.739267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.739179 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.739267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.739212 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.739267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.739243 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4q5\" (UniqueName: \"kubernetes.io/projected/9a561f60-a65e-4eeb-b7b2-b832818dc2bc-kube-api-access-xb4q5\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:33.908522 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.908485 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2"] Nov 03 08:23:33.916197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:33.916162 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/distinct-2-w4-15a633-kserve-7fc487d6cc-w89z2"] Nov 03 08:23:34.740587 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:34.740553 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5472f604-f001-48f8-a95a-4680d038e507" path="/var/lib/kubelet/pods/5472f604-f001-48f8-a95a-4680d038e507/volumes" Nov 03 08:23:34.741239 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:34.741221 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" path="/var/lib/kubelet/pods/9a561f60-a65e-4eeb-b7b2-b832818dc2bc/volumes" Nov 03 08:23:36.240603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:36.240526 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerName="payload-pre-processing" containerID="cri-o://400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423" gracePeriod=25 Nov 03 08:23:40.834614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:40.834559 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" containerID="cri-o://f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766" gracePeriod=45 Nov 03 08:23:40.853231 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:40.853181 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" containerID="cri-o://a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7" gracePeriod=45 Nov 03 08:23:40.857840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:40.857801 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" containerID="cri-o://386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0" gracePeriod=45 Nov 03 08:23:41.213342 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.212265 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:23:41.251540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.251511 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:23:41.257000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.256967 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:23:41.347617 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.347579 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir\") pod \"b310d0c8-f990-466f-b5bf-9a388124ef84\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " Nov 03 08:23:41.347843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.347808 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache\") pod \"b310d0c8-f990-466f-b5bf-9a388124ef84\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " Nov 03 08:23:41.347932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.347858 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir" pod "b310d0c8-f990-466f-b5bf-9a388124ef84" (UID: "b310d0c8-f990-466f-b5bf-9a388124ef84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.347932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.347903 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs\") pod \"cedc1ead-9210-438f-b524-aebf4b93942e\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " Nov 03 08:23:41.348053 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.347959 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8\" (UniqueName: \"kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8\") pod \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " Nov 03 08:23:41.348110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348053 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache\") pod \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " Nov 03 08:23:41.348110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348083 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm\") pod \"cedc1ead-9210-438f-b524-aebf4b93942e\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " Nov 03 08:23:41.348218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348142 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir\") pod \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " Nov 03 08:23:41.348218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348188 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home\") pod \"b310d0c8-f990-466f-b5bf-9a388124ef84\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " Nov 03 08:23:41.348348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348234 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir\") pod \"cedc1ead-9210-438f-b524-aebf4b93942e\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " Nov 03 08:23:41.348348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348270 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46\" (UniqueName: \"kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46\") pod \"b310d0c8-f990-466f-b5bf-9a388124ef84\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " Nov 03 08:23:41.348348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348316 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm\") pod \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " Nov 03 08:23:41.348515 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348351 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb\" (UniqueName: \"kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb\") pod \"cedc1ead-9210-438f-b524-aebf4b93942e\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " Nov 03 08:23:41.348515 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348381 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache\") pod \"cedc1ead-9210-438f-b524-aebf4b93942e\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " Nov 03 08:23:41.348515 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348449 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm\") pod \"b310d0c8-f990-466f-b5bf-9a388124ef84\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " Nov 03 08:23:41.348515 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348480 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home\") pod \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " Nov 03 08:23:41.348748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348576 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs\" (UniqueName: \"kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs\") pod \"b310d0c8-f990-466f-b5bf-9a388124ef84\" (UID: \"b310d0c8-f990-466f-b5bf-9a388124ef84\") " Nov 03 08:23:41.348748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348627 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs\") pod \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\" (UID: \"7d3588f6-d741-4e54-80dc-1ab1caa689ad\") " Nov 03 08:23:41.348859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.348820 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home\") pod \"cedc1ead-9210-438f-b524-aebf4b93942e\" (UID: \"cedc1ead-9210-438f-b524-aebf4b93942e\") " Nov 03 08:23:41.349840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.349815 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.349976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.349919 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache" pod "b310d0c8-f990-466f-b5bf-9a388124ef84" (UID: "b310d0c8-f990-466f-b5bf-9a388124ef84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.350045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.350021 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home" pod "cedc1ead-9210-438f-b524-aebf4b93942e" (UID: "cedc1ead-9210-438f-b524-aebf4b93942e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.350223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.350207 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home" pod "7d3588f6-d741-4e54-80dc-1ab1caa689ad" (UID: "7d3588f6-d741-4e54-80dc-1ab1caa689ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.352830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.352802 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache" pod "cedc1ead-9210-438f-b524-aebf4b93942e" (UID: "cedc1ead-9210-438f-b524-aebf4b93942e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.353017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.352993 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home" pod "b310d0c8-f990-466f-b5bf-9a388124ef84" (UID: "b310d0c8-f990-466f-b5bf-9a388124ef84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.353179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.353161 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir" pod "7d3588f6-d741-4e54-80dc-1ab1caa689ad" (UID: "7d3588f6-d741-4e54-80dc-1ab1caa689ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.355857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.355794 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir" pod "cedc1ead-9210-438f-b524-aebf4b93942e" (UID: "cedc1ead-9210-438f-b524-aebf4b93942e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.355995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.355938 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache" pod "7d3588f6-d741-4e54-80dc-1ab1caa689ad" (UID: "7d3588f6-d741-4e54-80dc-1ab1caa689ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.356447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.356362 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm" pod "7d3588f6-d741-4e54-80dc-1ab1caa689ad" (UID: "7d3588f6-d741-4e54-80dc-1ab1caa689ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.356447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.356423 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm" pod "cedc1ead-9210-438f-b524-aebf4b93942e" (UID: "cedc1ead-9210-438f-b524-aebf4b93942e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.357718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.357666 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb" pod "cedc1ead-9210-438f-b524-aebf4b93942e" (UID: "cedc1ead-9210-438f-b524-aebf4b93942e"). InnerVolumeSpecName "kube-api-access-lbbcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:41.358027 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.358009 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs" pod "7d3588f6-d741-4e54-80dc-1ab1caa689ad" (UID: "7d3588f6-d741-4e54-80dc-1ab1caa689ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:41.359016 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.358932 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46" pod "b310d0c8-f990-466f-b5bf-9a388124ef84" (UID: "b310d0c8-f990-466f-b5bf-9a388124ef84"). InnerVolumeSpecName "kube-api-access-xvn46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:41.359016 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.358999 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs" pod "b310d0c8-f990-466f-b5bf-9a388124ef84" (UID: "b310d0c8-f990-466f-b5bf-9a388124ef84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:41.359125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.359032 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8" pod "7d3588f6-d741-4e54-80dc-1ab1caa689ad" (UID: "7d3588f6-d741-4e54-80dc-1ab1caa689ad"). InnerVolumeSpecName "kube-api-access-mn7l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:41.359306 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.359283 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs" pod "cedc1ead-9210-438f-b524-aebf4b93942e" (UID: "cedc1ead-9210-438f-b524-aebf4b93942e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:41.360899 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.360629 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm" pod "b310d0c8-f990-466f-b5bf-9a388124ef84" (UID: "b310d0c8-f990-466f-b5bf-9a388124ef84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:41.451773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451736 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.451773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451768 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.451773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451781 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cedc1ead-9210-438f-b524-aebf4b93942e-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451791 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7l8\" (UniqueName: \"kubernetes.io/projected/7d3588f6-d741-4e54-80dc-1ab1caa689ad-kube-api-access-mn7l8\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451801 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451809 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451819 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451827 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451834 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451842 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvn46\" (UniqueName: \"kubernetes.io/projected/b310d0c8-f990-466f-b5bf-9a388124ef84-kube-api-access-xvn46\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451864 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451876 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbcb\" (UniqueName: \"kubernetes.io/projected/cedc1ead-9210-438f-b524-aebf4b93942e-kube-api-access-lbbcb\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451886 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cedc1ead-9210-438f-b524-aebf4b93942e-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451897 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b310d0c8-f990-466f-b5bf-9a388124ef84-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451905 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3588f6-d741-4e54-80dc-1ab1caa689ad-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451913 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b310d0c8-f990-466f-b5bf-9a388124ef84-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.452181 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.451922 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3588f6-d741-4e54-80dc-1ab1caa689ad-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:41.679361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.679278 5951 generic.go:356] "Generic (PLEG): container finished" podID="cedc1ead-9210-438f-b524-aebf4b93942e" containerID="f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766" exitCode=0 Nov 03 08:23:41.679361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.679324 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" Nov 03 08:23:41.679607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.679361 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" event={"ID":"cedc1ead-9210-438f-b524-aebf4b93942e","Type":"ContainerDied","Data":"f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766"} Nov 03 08:23:41.679607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.679409 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq" event={"ID":"cedc1ead-9210-438f-b524-aebf4b93942e","Type":"ContainerDied","Data":"cf40f93fb3643e1acc8e292d89c1023dd6e16669d643d911eb440c0c44d7ea89"} Nov 03 08:23:41.679607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.679426 5951 scope.go:122] "RemoveContainer" containerID="f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766" Nov 03 08:23:41.689773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.689737 5951 generic.go:356] "Generic (PLEG): container finished" podID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerID="386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0" exitCode=0 Nov 03 08:23:41.690094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.690071 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" event={"ID":"7d3588f6-d741-4e54-80dc-1ab1caa689ad","Type":"ContainerDied","Data":"386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0"} Nov 03 08:23:41.690259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.690243 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" event={"ID":"7d3588f6-d741-4e54-80dc-1ab1caa689ad","Type":"ContainerDied","Data":"602aaea9a23f7e9b25ecbc87b6d9db26de0d3d63adda80c3bca22bb142f3c032"} Nov 03 08:23:41.690525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.690506 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" Nov 03 08:23:41.693250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.693219 5951 generic.go:356] "Generic (PLEG): container finished" podID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerID="a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7" exitCode=0 Nov 03 08:23:41.693625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.693601 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" event={"ID":"b310d0c8-f990-466f-b5bf-9a388124ef84","Type":"ContainerDied","Data":"a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7"} Nov 03 08:23:41.693748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.693674 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" event={"ID":"b310d0c8-f990-466f-b5bf-9a388124ef84","Type":"ContainerDied","Data":"7cb4f0f3411eb7703e0d5684b64dcb5f5b1b9766b0bb7e63b2b4233cf2ec1769"} Nov 03 08:23:41.693818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.693776 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx" Nov 03 08:23:41.712760 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.712734 5951 scope.go:122] "RemoveContainer" containerID="f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766" Nov 03 08:23:41.713169 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:41.713140 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766\": container with ID starting with f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766 not found: ID does not exist" containerID="f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766" Nov 03 08:23:41.713271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.713176 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766"} err="failed to get container status \"f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766\": rpc error: code = NotFound desc = could not find container \"f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766\": container with ID starting with f921bc9a047a6a53ce6eec3d97aee0959a451e24d03960e62652c5b9e89d6766 not found: ID does not exist" Nov 03 08:23:41.713271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.713199 5951 scope.go:122] "RemoveContainer" containerID="386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0" Nov 03 08:23:41.719428 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.719398 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq"] Nov 03 08:23:41.727570 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.727411 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-2-w0-76ef13-kserve-6f5b9db49b-vrnjq"] Nov 03 08:23:41.729838 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.729808 5951 scope.go:122] "RemoveContainer" containerID="386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0" Nov 03 08:23:41.730411 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:41.730381 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0\": container with ID starting with 386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0 not found: ID does not exist" containerID="386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0" Nov 03 08:23:41.730518 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.730419 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0"} err="failed to get container status \"386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0\": rpc error: code = NotFound desc = could not find container \"386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0\": container with ID starting with 386153666d0ddf9f0461cf3c1cec72b9116634a2790f667a2a182675d84638c0 not found: ID does not exist" Nov 03 08:23:41.730518 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.730436 5951 scope.go:122] "RemoveContainer" containerID="a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7" Nov 03 08:23:41.736251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.736208 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7"] Nov 03 08:23:41.744018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.743966 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7"] Nov 03 08:23:41.744440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.744420 5951 scope.go:122] "RemoveContainer" containerID="a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7" Nov 03 08:23:41.744864 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:41.744838 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7\": container with ID starting with a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7 not found: ID does not exist" containerID="a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7" Nov 03 08:23:41.744952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.744877 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7"} err="failed to get container status \"a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7\": rpc error: code = NotFound desc = could not find container \"a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7\": container with ID starting with a2181c256d488f69688610f16348ec9c062bd67a29f1be4d6b263a5761326cd7 not found: ID does not exist" Nov 03 08:23:41.748538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.748517 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx"] Nov 03 08:23:41.753045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:41.753023 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-76ef13/distinct-w0-76ef13-kserve-8595ddffc9-v7dsx"] Nov 03 08:23:42.144671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:42.142432 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-76ef13/premium-w0-76ef13-kserve-dd498c768-znlf7" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:23:42.811889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:42.811847 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" path="/var/lib/kubelet/pods/7d3588f6-d741-4e54-80dc-1ab1caa689ad/volumes" Nov 03 08:23:42.812593 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:42.812570 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" path="/var/lib/kubelet/pods/b310d0c8-f990-466f-b5bf-9a388124ef84/volumes" Nov 03 08:23:42.813605 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:42.813383 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" path="/var/lib/kubelet/pods/cedc1ead-9210-438f-b524-aebf4b93942e/volumes" Nov 03 08:23:44.673479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:44.673442 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:23:47.362440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:47.362410 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt"] Nov 03 08:23:48.664758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:48.664718 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4"] Nov 03 08:23:48.683125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:48.683052 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp"] Nov 03 08:23:52.363630 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:52.363572 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerName="payload-pre-processing" containerID="cri-o://3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc" gracePeriod=25 Nov 03 08:23:53.484689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:53.484631 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6"] Nov 03 08:23:53.485189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:53.484941 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" containerID="cri-o://75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510" gracePeriod=30 Nov 03 08:23:53.665758 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:53.665694 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" podUID="985bd2be-13e7-40a7-9022-38979241bc88" containerName="payload-processing" containerID="cri-o://feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913" gracePeriod=25 Nov 03 08:23:53.686383 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:53.685532 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerName="payload-pre-processing" containerID="cri-o://f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af" gracePeriod=25 Nov 03 08:23:54.209172 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:54.209137 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:23:54.673542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:54.673502 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:23:55.902406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:55.902371 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k"] Nov 03 08:23:55.904116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:55.904083 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" podUID="388d8926-1579-4394-aa5f-fed4bee26395" containerName="maas-api" containerID="cri-o://92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d" gracePeriod=30 Nov 03 08:23:56.051303 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.051261 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq"] Nov 03 08:23:56.051723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.051692 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" podUID="0ca589f5-8f4e-48b6-8642-44d214a72d28" containerName="maas-api" containerID="cri-o://58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb" gracePeriod=30 Nov 03 08:23:56.663082 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.663049 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:23:56.714285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.713839 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls\") pod \"0ca589f5-8f4e-48b6-8642-44d214a72d28\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " Nov 03 08:23:56.714934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.714821 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs\") pod \"0ca589f5-8f4e-48b6-8642-44d214a72d28\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " Nov 03 08:23:56.715157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.715107 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt\" (UniqueName: \"kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt\") pod \"0ca589f5-8f4e-48b6-8642-44d214a72d28\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " Nov 03 08:23:56.715554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.715515 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca\") pod \"0ca589f5-8f4e-48b6-8642-44d214a72d28\" (UID: \"0ca589f5-8f4e-48b6-8642-44d214a72d28\") " Nov 03 08:23:56.717269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.717017 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca" pod "0ca589f5-8f4e-48b6-8642-44d214a72d28" (UID: "0ca589f5-8f4e-48b6-8642-44d214a72d28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:56.738507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.737258 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls" pod "0ca589f5-8f4e-48b6-8642-44d214a72d28" (UID: "0ca589f5-8f4e-48b6-8642-44d214a72d28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:56.747997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.742241 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt" pod "0ca589f5-8f4e-48b6-8642-44d214a72d28" (UID: "0ca589f5-8f4e-48b6-8642-44d214a72d28"). InnerVolumeSpecName "kube-api-access-nz6gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:56.749296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.749250 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs" pod "0ca589f5-8f4e-48b6-8642-44d214a72d28" (UID: "0ca589f5-8f4e-48b6-8642-44d214a72d28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:56.820047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.819746 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:56.820047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.819777 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6gt\" (UniqueName: \"kubernetes.io/projected/0ca589f5-8f4e-48b6-8642-44d214a72d28-kube-api-access-nz6gt\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:56.820047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.819791 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca589f5-8f4e-48b6-8642-44d214a72d28-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:56.820047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.819806 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca589f5-8f4e-48b6-8642-44d214a72d28-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:56.835624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.835182 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:23:56.913670 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.911764 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:23:56.921259 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.921210 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca\") pod \"388d8926-1579-4394-aa5f-fed4bee26395\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " Nov 03 08:23:56.921443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.921406 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\") pod \"388d8926-1579-4394-aa5f-fed4bee26395\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " Nov 03 08:23:56.921831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.921806 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs\") pod \"388d8926-1579-4394-aa5f-fed4bee26395\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " Nov 03 08:23:56.922047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.922029 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2\" (UniqueName: \"kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2\") pod \"388d8926-1579-4394-aa5f-fed4bee26395\" (UID: \"388d8926-1579-4394-aa5f-fed4bee26395\") " Nov 03 08:23:56.935749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.935433 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca" pod "388d8926-1579-4394-aa5f-fed4bee26395" (UID: "388d8926-1579-4394-aa5f-fed4bee26395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:56.940323 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.940077 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2" pod "388d8926-1579-4394-aa5f-fed4bee26395" (UID: "388d8926-1579-4394-aa5f-fed4bee26395"). InnerVolumeSpecName "kube-api-access-ktfj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:56.940323 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.940181 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs" pod "388d8926-1579-4394-aa5f-fed4bee26395" (UID: "388d8926-1579-4394-aa5f-fed4bee26395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:56.946668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:56.943087 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls" pod "388d8926-1579-4394-aa5f-fed4bee26395" (UID: "388d8926-1579-4394-aa5f-fed4bee26395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:23:57.037293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.037258 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume\") pod \"b87dd5f9-9463-42d1-89eb-91c0c287521a\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " Nov 03 08:23:57.037537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.037523 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4\" (UniqueName: \"kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4\") pod \"b87dd5f9-9463-42d1-89eb-91c0c287521a\" (UID: \"b87dd5f9-9463-42d1-89eb-91c0c287521a\") " Nov 03 08:23:57.037785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.037756 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume" pod "b87dd5f9-9463-42d1-89eb-91c0c287521a" (UID: "b87dd5f9-9463-42d1-89eb-91c0c287521a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:57.038294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.038269 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:57.038410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.038299 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfj2\" (UniqueName: \"kubernetes.io/projected/388d8926-1579-4394-aa5f-fed4bee26395-kube-api-access-ktfj2\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:57.038410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.038317 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388d8926-1579-4394-aa5f-fed4bee26395-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:57.038410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.038333 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87dd5f9-9463-42d1-89eb-91c0c287521a-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:57.038410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.038347 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388d8926-1579-4394-aa5f-fed4bee26395-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:57.040140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.040110 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4" pod "b87dd5f9-9463-42d1-89eb-91c0c287521a" (UID: "b87dd5f9-9463-42d1-89eb-91c0c287521a"). InnerVolumeSpecName "kube-api-access-j9dj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:57.140380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.140350 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dj4\" (UniqueName: \"kubernetes.io/projected/b87dd5f9-9463-42d1-89eb-91c0c287521a-kube-api-access-j9dj4\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:57.162034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.161995 5951 generic.go:356] "Generic (PLEG): container finished" podID="0ca589f5-8f4e-48b6-8642-44d214a72d28" containerID="58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb" exitCode=0 Nov 03 08:23:57.162220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.162043 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" Nov 03 08:23:57.162220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.162084 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" event={"ID":"0ca589f5-8f4e-48b6-8642-44d214a72d28","Type":"ContainerDied","Data":"58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb"} Nov 03 08:23:57.162220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.162149 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq" event={"ID":"0ca589f5-8f4e-48b6-8642-44d214a72d28","Type":"ContainerDied","Data":"94edf48b9c7c1cb10eb5424404ec451a8b466498af34f5b386afc7784443a97a"} Nov 03 08:23:57.162220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.162180 5951 scope.go:122] "RemoveContainer" containerID="58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb" Nov 03 08:23:57.163852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.163802 5951 generic.go:356] "Generic (PLEG): container finished" podID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerID="400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423" exitCode=0 Nov 03 08:23:57.163916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.163858 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" Nov 03 08:23:57.163916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.163868 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" event={"ID":"b87dd5f9-9463-42d1-89eb-91c0c287521a","Type":"ContainerDied","Data":"400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423"} Nov 03 08:23:57.164024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.163917 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l" event={"ID":"b87dd5f9-9463-42d1-89eb-91c0c287521a","Type":"ContainerDied","Data":"efbede0afeb96a67b1b5121645fd195f36be06a80b029514620cadc5f23c9237"} Nov 03 08:23:57.165745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.165720 5951 generic.go:356] "Generic (PLEG): container finished" podID="388d8926-1579-4394-aa5f-fed4bee26395" containerID="92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d" exitCode=0 Nov 03 08:23:57.165857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.165755 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" Nov 03 08:23:57.165857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.165800 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" event={"ID":"388d8926-1579-4394-aa5f-fed4bee26395","Type":"ContainerDied","Data":"92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d"} Nov 03 08:23:57.165857 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.165842 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k" event={"ID":"388d8926-1579-4394-aa5f-fed4bee26395","Type":"ContainerDied","Data":"9e9a50b91034d21bdb940622be303edd6c99b8519daf9950a3c240cdabe6ac07"} Nov 03 08:23:57.175449 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.175400 5951 scope.go:122] "RemoveContainer" containerID="58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb" Nov 03 08:23:57.175798 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:57.175773 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb\": container with ID starting with 58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb not found: ID does not exist" containerID="58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb" Nov 03 08:23:57.175891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.175808 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb"} err="failed to get container status \"58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb\": rpc error: code = NotFound desc = could not find container \"58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb\": container with ID starting with 58bf8442db68698dd6b6a9d6c524a8980af50c6e56b86e15509a97e5d3b3f4fb not found: ID does not exist" Nov 03 08:23:57.175891 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.175829 5951 scope.go:122] "RemoveContainer" containerID="400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423" Nov 03 08:23:57.189613 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.189579 5951 scope.go:122] "RemoveContainer" containerID="400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423" Nov 03 08:23:57.190136 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:57.190103 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423\": container with ID starting with 400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423 not found: ID does not exist" containerID="400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423" Nov 03 08:23:57.190136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.190131 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423"} err="failed to get container status \"400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423\": rpc error: code = NotFound desc = could not find container \"400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423\": container with ID starting with 400d3734b2cd38d71381b21ca6bf70cc9fdfe2a32585321f41113a15cdfd2423 not found: ID does not exist" Nov 03 08:23:57.190356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.190148 5951 scope.go:122] "RemoveContainer" containerID="92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d" Nov 03 08:23:57.193579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.193557 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq"] Nov 03 08:23:57.199304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.199279 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-15a633-7d64544846-tjczq"] Nov 03 08:23:57.202995 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.202976 5951 scope.go:122] "RemoveContainer" containerID="92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d" Nov 03 08:23:57.203403 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:57.203383 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d\": container with ID starting with 92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d not found: ID does not exist" containerID="92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d" Nov 03 08:23:57.203487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.203413 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d"} err="failed to get container status \"92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d\": rpc error: code = NotFound desc = could not find container \"92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d\": container with ID starting with 92016e7d0f5c73711e25cb8870fd1171997e375704f47ee46facba5afcb6e94d not found: ID does not exist" Nov 03 08:23:57.208045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.207315 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k"] Nov 03 08:23:57.213316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.213296 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e0f576-5745cbc89c-nw48k"] Nov 03 08:23:57.225860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.225837 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l"] Nov 03 08:23:57.231599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:57.231575 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-76ef13-58f4965855-x8q6l"] Nov 03 08:23:58.743507 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.743460 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca589f5-8f4e-48b6-8642-44d214a72d28" path="/var/lib/kubelet/pods/0ca589f5-8f4e-48b6-8642-44d214a72d28/volumes" Nov 03 08:23:58.744161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.744141 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388d8926-1579-4394-aa5f-fed4bee26395" path="/var/lib/kubelet/pods/388d8926-1579-4394-aa5f-fed4bee26395/volumes" Nov 03 08:23:58.744568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.744544 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" path="/var/lib/kubelet/pods/b87dd5f9-9463-42d1-89eb-91c0c287521a/volumes" Nov 03 08:23:58.808388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.808017 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:23:58.884174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884145 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884218 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884285 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884317 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884366 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884488 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884547 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884533 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:58.884756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884581 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884756 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884738 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.884875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.884801 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs\") pod \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\" (UID: \"74eefb2e-d52d-42f9-b10b-10f7efa5c038\") " Nov 03 08:23:58.885146 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885116 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:58.885244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885140 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:23:58.885244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885204 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:58.885582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885505 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:58.885947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885920 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.886040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885953 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.886040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885968 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.886040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885982 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.886040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.885996 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.887563 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.887539 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:23:58.887691 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.887589 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "kube-api-access-s9qnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:58.887766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.887747 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:23:58.888188 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.888159 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token" pod "74eefb2e-d52d-42f9-b10b-10f7efa5c038" (UID: "74eefb2e-d52d-42f9-b10b-10f7efa5c038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:23:58.987929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.987900 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.987929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.987926 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9qnb\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-kube-api-access-s9qnb\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.987929 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.987936 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:58.988157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:58.987945 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eefb2e-d52d-42f9-b10b-10f7efa5c038-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:23:59.186719 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.186678 5951 generic.go:356] "Generic (PLEG): container finished" podID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerID="75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510" exitCode=0 Nov 03 08:23:59.186917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.186735 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" Nov 03 08:23:59.186917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.186748 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" event={"ID":"74eefb2e-d52d-42f9-b10b-10f7efa5c038","Type":"ContainerDied","Data":"75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510"} Nov 03 08:23:59.186917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.186820 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6" event={"ID":"74eefb2e-d52d-42f9-b10b-10f7efa5c038","Type":"ContainerDied","Data":"0685879cf0823bc82021212eb856988ac66de9f8d66ebb6195d1013848a2b4bf"} Nov 03 08:23:59.186917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.186852 5951 scope.go:122] "RemoveContainer" containerID="75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510" Nov 03 08:23:59.202423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.202402 5951 scope.go:122] "RemoveContainer" containerID="75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510" Nov 03 08:23:59.202971 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:23:59.202944 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510\": container with ID starting with 75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510 not found: ID does not exist" containerID="75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510" Nov 03 08:23:59.203108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.202984 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510"} err="failed to get container status \"75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510\": rpc error: code = NotFound desc = could not find container \"75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510\": container with ID starting with 75f9f2cb91806ed7ca2c86fd970acd22a9ff966bb77c6152018e713b79344510 not found: ID does not exist" Nov 03 08:23:59.213258 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.213229 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6"] Nov 03 08:23:59.217294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:23:59.217267 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-726m6"] Nov 03 08:24:00.742991 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:00.742961 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" path="/var/lib/kubelet/pods/74eefb2e-d52d-42f9-b10b-10f7efa5c038/volumes" Nov 03 08:24:01.957406 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:01.957370 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:24:01.968775 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:01.968733 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:01.968950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:01.968927 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" podUID="985bd2be-13e7-40a7-9022-38979241bc88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:02.636385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:02.636349 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj"] Nov 03 08:24:02.672909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:02.672860 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" podUID="f15ad4bc-6867-4027-b055-bf9b62894b9a" containerName="maas-api" containerID="cri-o://9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e" gracePeriod=30 Nov 03 08:24:02.683772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:02.682864 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f"] Nov 03 08:24:02.683772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:02.683289 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" containerID="cri-o://7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861" gracePeriod=30 Nov 03 08:24:02.697790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:02.696253 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt"] Nov 03 08:24:02.697790 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:02.696491 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" containerID="cri-o://3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9" gracePeriod=30 Nov 03 08:24:03.254709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.251362 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:24:03.265559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.264547 5951 generic.go:356] "Generic (PLEG): container finished" podID="f15ad4bc-6867-4027-b055-bf9b62894b9a" containerID="9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e" exitCode=0 Nov 03 08:24:03.265559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.264623 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" event={"ID":"f15ad4bc-6867-4027-b055-bf9b62894b9a","Type":"ContainerDied","Data":"9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e"} Nov 03 08:24:03.265559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.264720 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" event={"ID":"f15ad4bc-6867-4027-b055-bf9b62894b9a","Type":"ContainerDied","Data":"236559920abfcc8d63df5e808fbf99f327c01a382c4a01e6750fc929cfea8a2c"} Nov 03 08:24:03.265559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.264940 5951 scope.go:122] "RemoveContainer" containerID="9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e" Nov 03 08:24:03.265559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.265011 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj" Nov 03 08:24:03.289713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.289690 5951 scope.go:122] "RemoveContainer" containerID="9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e" Nov 03 08:24:03.290360 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:03.290316 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e\": container with ID starting with 9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e not found: ID does not exist" containerID="9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e" Nov 03 08:24:03.290549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.290515 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e"} err="failed to get container status \"9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e\": rpc error: code = NotFound desc = could not find container \"9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e\": container with ID starting with 9dda7a361f623c8d35e41b5c8b2658dce21170eb6a41ca5171f840637d47cf9e not found: ID does not exist" Nov 03 08:24:03.393399 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.393359 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs\") pod \"f15ad4bc-6867-4027-b055-bf9b62894b9a\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " Nov 03 08:24:03.393607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.393534 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f\" (UniqueName: \"kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f\") pod \"f15ad4bc-6867-4027-b055-bf9b62894b9a\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " Nov 03 08:24:03.395262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.393776 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\") pod \"f15ad4bc-6867-4027-b055-bf9b62894b9a\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " Nov 03 08:24:03.395262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.393815 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca\") pod \"f15ad4bc-6867-4027-b055-bf9b62894b9a\" (UID: \"f15ad4bc-6867-4027-b055-bf9b62894b9a\") " Nov 03 08:24:03.402876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.401904 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca" pod "f15ad4bc-6867-4027-b055-bf9b62894b9a" (UID: "f15ad4bc-6867-4027-b055-bf9b62894b9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:03.419284 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.419198 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs" pod "f15ad4bc-6867-4027-b055-bf9b62894b9a" (UID: "f15ad4bc-6867-4027-b055-bf9b62894b9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:24:03.419940 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.419914 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls" pod "f15ad4bc-6867-4027-b055-bf9b62894b9a" (UID: "f15ad4bc-6867-4027-b055-bf9b62894b9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:24:03.422344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.422303 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f" pod "f15ad4bc-6867-4027-b055-bf9b62894b9a" (UID: "f15ad4bc-6867-4027-b055-bf9b62894b9a"). InnerVolumeSpecName "kube-api-access-t756f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:03.495846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.495753 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:03.495846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.495780 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t756f\" (UniqueName: \"kubernetes.io/projected/f15ad4bc-6867-4027-b055-bf9b62894b9a-kube-api-access-t756f\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:03.495846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.495789 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15ad4bc-6867-4027-b055-bf9b62894b9a-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:03.495846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.495800 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15ad4bc-6867-4027-b055-bf9b62894b9a-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:03.588428 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.588397 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj"] Nov 03 08:24:03.595956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:03.595928 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b559e-855f45fcdc-j9zkj"] Nov 03 08:24:04.213174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:04.213123 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:04.492305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:04.483863 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:24:04.492305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:04.483921 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:24:04.739018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:04.738976 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15ad4bc-6867-4027-b055-bf9b62894b9a" path="/var/lib/kubelet/pods/f15ad4bc-6867-4027-b055-bf9b62894b9a/volumes" Nov 03 08:24:04.852883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:04.852845 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45"] Nov 03 08:24:04.853107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:04.853082 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" containerID="cri-o://ddfd53388cdf1675e5639ae62a6366e1a37840143924922df23636f3e197139b" gracePeriod=30 Nov 03 08:24:05.570572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:05.570535 5951 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:24:05.571091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:05.570617 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:24:08.153679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.150815 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:24:08.211249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.211215 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:24:08.217354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217320 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217394 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217542 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217487 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217543 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217683 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217819 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217731 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217901 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217871 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.217989 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.217971 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.219170 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.219133 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket\") pod \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\" (UID: \"fd5e3cdd-09d6-40d8-b479-1179b58a9c27\") " Nov 03 08:24:08.220105 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.218247 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.220277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.218292 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.220277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.218819 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:08.220277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.219032 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.220541 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.220524 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.221427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.221387 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:08.223069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.223043 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "kube-api-access-tkpxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:08.223424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.223402 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.226317 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.226281 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token" pod "fd5e3cdd-09d6-40d8-b479-1179b58a9c27" (UID: "fd5e3cdd-09d6-40d8-b479-1179b58a9c27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:08.316862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.316824 5951 generic.go:356] "Generic (PLEG): container finished" podID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerID="7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861" exitCode=0 Nov 03 08:24:08.317043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.316923 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" event={"ID":"fd5e3cdd-09d6-40d8-b479-1179b58a9c27","Type":"ContainerDied","Data":"7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861"} Nov 03 08:24:08.317194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.317039 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" event={"ID":"fd5e3cdd-09d6-40d8-b479-1179b58a9c27","Type":"ContainerDied","Data":"af9535f52ad7b76d8f85f2564787fb14e5190a4aac60d359b0553d8540b57a5a"} Nov 03 08:24:08.317194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.317096 5951 scope.go:122] "RemoveContainer" containerID="7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861" Nov 03 08:24:08.317332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.317277 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f" Nov 03 08:24:08.320119 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320087 5951 generic.go:356] "Generic (PLEG): container finished" podID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerID="3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9" exitCode=0 Nov 03 08:24:08.320229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320134 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" event={"ID":"c7615b3e-68c7-4e1d-9230-30be0de2d060","Type":"ContainerDied","Data":"3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9"} Nov 03 08:24:08.320229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320193 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" event={"ID":"c7615b3e-68c7-4e1d-9230-30be0de2d060","Type":"ContainerDied","Data":"adbcdb38381f58e94c2e1ba06974fdb6922315b3ed7b8616f2cfed30e55b6a0e"} Nov 03 08:24:08.320352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320280 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt" Nov 03 08:24:08.320412 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320380 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.320471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320450 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.320525 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320482 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.320581 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320547 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.320658 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320580 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.321073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320815 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.321073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320891 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.321073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.320964 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.321476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.321418 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert\") pod \"c7615b3e-68c7-4e1d-9230-30be0de2d060\" (UID: \"c7615b3e-68c7-4e1d-9230-30be0de2d060\") " Nov 03 08:24:08.322057 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.321933 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.322706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.322198 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.322706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.322452 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.322966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.322932 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.323955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.323919 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325109 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325136 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325152 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325165 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325179 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7615b3e-68c7-4e1d-9230-30be0de2d060-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325192 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325206 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325220 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325234 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325247 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325261 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325275 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkpxk\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-kube-api-access-tkpxk\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325291 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.325330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.325304 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5e3cdd-09d6-40d8-b479-1179b58a9c27-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.326581 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.326540 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:08.333089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.333042 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:08.334984 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.334950 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "kube-api-access-ntffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:08.335436 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.335411 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy" pod "c7615b3e-68c7-4e1d-9230-30be0de2d060" (UID: "c7615b3e-68c7-4e1d-9230-30be0de2d060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:08.339757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.339736 5951 scope.go:122] "RemoveContainer" containerID="7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861" Nov 03 08:24:08.340380 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:08.340250 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861\": container with ID starting with 7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861 not found: ID does not exist" containerID="7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861" Nov 03 08:24:08.340380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.340289 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861"} err="failed to get container status \"7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861\": rpc error: code = NotFound desc = could not find container \"7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861\": container with ID starting with 7dedbec68ba0f0f6f9063c21d1af4b3f8c0515391d99b1769ef534093045b861 not found: ID does not exist" Nov 03 08:24:08.340380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.340315 5951 scope.go:122] "RemoveContainer" containerID="3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9" Nov 03 08:24:08.354229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.354210 5951 scope.go:122] "RemoveContainer" containerID="3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9" Nov 03 08:24:08.354612 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:08.354585 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9\": container with ID starting with 3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9 not found: ID does not exist" containerID="3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9" Nov 03 08:24:08.354736 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.354629 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9"} err="failed to get container status \"3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9\": rpc error: code = NotFound desc = could not find container \"3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9\": container with ID starting with 3f90f637aad4fd0ea595238d065aed5fef5a57026a3d618be345e8a24d7891e9 not found: ID does not exist" Nov 03 08:24:08.360722 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.360598 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f"] Nov 03 08:24:08.364701 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.364672 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-4zc8f"] Nov 03 08:24:08.426624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.426584 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntffg\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-kube-api-access-ntffg\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.426624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.426615 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.426624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.426626 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.426905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.426634 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7615b3e-68c7-4e1d-9230-30be0de2d060-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:08.646455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.646421 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt"] Nov 03 08:24:08.653073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.653042 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-5qprt"] Nov 03 08:24:08.742577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.742540 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" path="/var/lib/kubelet/pods/c7615b3e-68c7-4e1d-9230-30be0de2d060/volumes" Nov 03 08:24:08.743111 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:08.743093 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" path="/var/lib/kubelet/pods/fd5e3cdd-09d6-40d8-b479-1179b58a9c27/volumes" Nov 03 08:24:09.078953 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:09.078915 5951 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:24:09.079279 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:09.079237 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:24:10.076987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.076716 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl"] Nov 03 08:24:10.076987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.076972 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" containerID="cri-o://bc9698e21c4b95531d254f9d04c75fa609d59dfa93d1251bbe743b4a3ce9b0cc" gracePeriod=30 Nov 03 08:24:10.084860 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.084833 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw"] Nov 03 08:24:10.085253 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.085224 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" containerID="cri-o://ff45e8208f0978021b9b2563eef9723934a2a35e727e1b9d3dec5c6c8d4b6be1" gracePeriod=30 Nov 03 08:24:10.088595 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.088571 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84"] Nov 03 08:24:10.088814 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.088787 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" containerID="cri-o://e8a1dc870c69bdcdbde7a5642d218c9c64ff9d23b9f909b22e8958fbe9409252" gracePeriod=30 Nov 03 08:24:10.434458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.434377 5951 generic.go:356] "Generic (PLEG): container finished" podID="ed23459d-9aa8-492e-872d-5676cab351b4" containerID="ddfd53388cdf1675e5639ae62a6366e1a37840143924922df23636f3e197139b" exitCode=0 Nov 03 08:24:10.436328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.436224 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" event={"ID":"ed23459d-9aa8-492e-872d-5676cab351b4","Type":"ContainerDied","Data":"ddfd53388cdf1675e5639ae62a6366e1a37840143924922df23636f3e197139b"} Nov 03 08:24:10.446275 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.446249 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:24:10.487845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.487805 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.487858 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.487891 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488078 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:10.488184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488109 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488185 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488236 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488223 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:10.488454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488361 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488374 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:10.488454 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488396 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488466 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token\") pod \"ed23459d-9aa8-492e-872d-5676cab351b4\" (UID: \"ed23459d-9aa8-492e-872d-5676cab351b4\") " Nov 03 08:24:10.488611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488496 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:10.488794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.488771 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:10.489272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.489243 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed23459d-9aa8-492e-872d-5676cab351b4-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.489384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.489275 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.489384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.489291 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.489384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.489305 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.489384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.489317 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.491405 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.491382 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "kube-api-access-pxnzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:10.491954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.491912 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:10.491954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.491935 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:10.493256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.493235 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token" pod "ed23459d-9aa8-492e-872d-5676cab351b4" (UID: "ed23459d-9aa8-492e-872d-5676cab351b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:10.591251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.591215 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxnzk\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-kube-api-access-pxnzk\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.591251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.591247 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed23459d-9aa8-492e-872d-5676cab351b4-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.591251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.591256 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed23459d-9aa8-492e-872d-5676cab351b4-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:10.591478 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:10.591265 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed23459d-9aa8-492e-872d-5676cab351b4-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:11.447703 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.447670 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" Nov 03 08:24:11.448128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.447701 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45" event={"ID":"ed23459d-9aa8-492e-872d-5676cab351b4","Type":"ContainerDied","Data":"69b17c5e29cc06a7911d9c8de2a6eaf21a349105c7d1e909a992f63fefc5e118"} Nov 03 08:24:11.448128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.447766 5951 scope.go:122] "RemoveContainer" containerID="ddfd53388cdf1675e5639ae62a6366e1a37840143924922df23636f3e197139b" Nov 03 08:24:11.466330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.466299 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45"] Nov 03 08:24:11.473513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.473485 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-2fr45"] Nov 03 08:24:11.695667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.695615 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:24:11.695848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:11.695722 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:24:12.002889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.002843 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" podUID="985bd2be-13e7-40a7-9022-38979241bc88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:12.003333 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.003297 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:12.684413 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.684387 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:24:12.742469 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.742432 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" path="/var/lib/kubelet/pods/ed23459d-9aa8-492e-872d-5676cab351b4/volumes" Nov 03 08:24:12.806443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.803907 5951 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:24:12.806443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.803969 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:24:12.826852 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.826823 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26\" (UniqueName: \"kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26\") pod \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " Nov 03 08:24:12.827244 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.826953 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume\") pod \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\" (UID: \"5bb7caeb-48b0-4683-9dc2-08d96a124b64\") " Nov 03 08:24:12.827384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.827285 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume" pod "5bb7caeb-48b0-4683-9dc2-08d96a124b64" (UID: "5bb7caeb-48b0-4683-9dc2-08d96a124b64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:12.827875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.827854 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb7caeb-48b0-4683-9dc2-08d96a124b64-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:12.829238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.829216 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26" pod "5bb7caeb-48b0-4683-9dc2-08d96a124b64" (UID: "5bb7caeb-48b0-4683-9dc2-08d96a124b64"). InnerVolumeSpecName "kube-api-access-gqr26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:12.929264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:12.929185 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqr26\" (UniqueName: \"kubernetes.io/projected/5bb7caeb-48b0-4683-9dc2-08d96a124b64-kube-api-access-gqr26\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:13.458020 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.457988 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq"] Nov 03 08:24:13.475140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.475104 5951 generic.go:356] "Generic (PLEG): container finished" podID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerID="3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc" exitCode=0 Nov 03 08:24:13.475296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.475151 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" Nov 03 08:24:13.475296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.475160 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" event={"ID":"5bb7caeb-48b0-4683-9dc2-08d96a124b64","Type":"ContainerDied","Data":"3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc"} Nov 03 08:24:13.475296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.475227 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt" event={"ID":"5bb7caeb-48b0-4683-9dc2-08d96a124b64","Type":"ContainerDied","Data":"5ec97a962f2b2189bd8c263ae8d282f5b9b4a8261fc8dc00b7e202b40ecd1aa9"} Nov 03 08:24:13.475296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.475256 5951 scope.go:122] "RemoveContainer" containerID="3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc" Nov 03 08:24:13.493431 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.493403 5951 scope.go:122] "RemoveContainer" containerID="3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc" Nov 03 08:24:13.493915 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:13.493825 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc\": container with ID starting with 3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc not found: ID does not exist" containerID="3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc" Nov 03 08:24:13.493915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.493859 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc"} err="failed to get container status \"3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc\": rpc error: code = NotFound desc = could not find container \"3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc\": container with ID starting with 3a7227a4ad7f7624e5b854601085bee16d376c6ce14c51c9d04433cc2d97a8dc not found: ID does not exist" Nov 03 08:24:13.529761 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.526400 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt"] Nov 03 08:24:13.533679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.533059 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e0f576-84c9b98d4c-lcvkt"] Nov 03 08:24:13.690401 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.690367 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk"] Nov 03 08:24:13.690882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.690702 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" containerID="cri-o://fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5" gracePeriod=30 Nov 03 08:24:13.698177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.698136 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8"] Nov 03 08:24:13.698895 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:13.698672 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" containerID="cri-o://9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8" gracePeriod=30 Nov 03 08:24:14.086687 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.086556 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:24:14.165535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.165495 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume\") pod \"985bd2be-13e7-40a7-9022-38979241bc88\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " Nov 03 08:24:14.165713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.165573 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx\" (UniqueName: \"kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx\") pod \"985bd2be-13e7-40a7-9022-38979241bc88\" (UID: \"985bd2be-13e7-40a7-9022-38979241bc88\") " Nov 03 08:24:14.165997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.165969 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume" pod "985bd2be-13e7-40a7-9022-38979241bc88" (UID: "985bd2be-13e7-40a7-9022-38979241bc88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:14.166432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.166407 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985bd2be-13e7-40a7-9022-38979241bc88-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:14.168766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.168736 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx" pod "985bd2be-13e7-40a7-9022-38979241bc88" (UID: "985bd2be-13e7-40a7-9022-38979241bc88"). InnerVolumeSpecName "kube-api-access-7mdjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:14.220528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.220489 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:24:14.268254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.268208 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdjx\" (UniqueName: \"kubernetes.io/projected/985bd2be-13e7-40a7-9022-38979241bc88-kube-api-access-7mdjx\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:14.282900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.282863 5951 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:24:14.283089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.282940 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:24:14.369776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.369673 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume\") pod \"e9415064-28a2-41a2-bfb8-0a06634ab720\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " Nov 03 08:24:14.369943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.369804 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln\" (UniqueName: \"kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln\") pod \"e9415064-28a2-41a2-bfb8-0a06634ab720\" (UID: \"e9415064-28a2-41a2-bfb8-0a06634ab720\") " Nov 03 08:24:14.370125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.370101 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume" pod "e9415064-28a2-41a2-bfb8-0a06634ab720" (UID: "e9415064-28a2-41a2-bfb8-0a06634ab720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:14.370699 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.370630 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9415064-28a2-41a2-bfb8-0a06634ab720-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:14.372035 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.372013 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln" pod "e9415064-28a2-41a2-bfb8-0a06634ab720" (UID: "e9415064-28a2-41a2-bfb8-0a06634ab720"). InnerVolumeSpecName "kube-api-access-4l8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:14.472141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.472111 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8ln\" (UniqueName: \"kubernetes.io/projected/e9415064-28a2-41a2-bfb8-0a06634ab720-kube-api-access-4l8ln\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:14.488016 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.487966 5951 generic.go:356] "Generic (PLEG): container finished" podID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerID="f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af" exitCode=0 Nov 03 08:24:14.488203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.488093 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" event={"ID":"e9415064-28a2-41a2-bfb8-0a06634ab720","Type":"ContainerDied","Data":"f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af"} Nov 03 08:24:14.488203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.488146 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" event={"ID":"e9415064-28a2-41a2-bfb8-0a06634ab720","Type":"ContainerDied","Data":"2688215e04a9f1a5a1a284ec23bdd979a06dcce40f0d6b6c66443cafd420773b"} Nov 03 08:24:14.488203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.488175 5951 scope.go:122] "RemoveContainer" containerID="f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af" Nov 03 08:24:14.488359 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.488275 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp" Nov 03 08:24:14.498304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.498270 5951 generic.go:356] "Generic (PLEG): container finished" podID="985bd2be-13e7-40a7-9022-38979241bc88" containerID="feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913" exitCode=0 Nov 03 08:24:14.498439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.498392 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" event={"ID":"985bd2be-13e7-40a7-9022-38979241bc88","Type":"ContainerDied","Data":"feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913"} Nov 03 08:24:14.498520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.498488 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" event={"ID":"985bd2be-13e7-40a7-9022-38979241bc88","Type":"ContainerDied","Data":"aeac0b4acc782eb9b0b0832b96d7629a09d4977db076428185933de3b7d1fd4e"} Nov 03 08:24:14.498590 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.498570 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4" Nov 03 08:24:14.515355 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.515332 5951 scope.go:122] "RemoveContainer" containerID="f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af" Nov 03 08:24:14.517632 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:14.517591 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af\": container with ID starting with f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af not found: ID does not exist" containerID="f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af" Nov 03 08:24:14.517834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.517799 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af"} err="failed to get container status \"f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af\": rpc error: code = NotFound desc = could not find container \"f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af\": container with ID starting with f6b2713a6bed681e296ede1e3af607c7c03707cfeda474f64a0e89d8ae3156af not found: ID does not exist" Nov 03 08:24:14.517834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.517835 5951 scope.go:122] "RemoveContainer" containerID="feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913" Nov 03 08:24:14.536753 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.536722 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4"] Nov 03 08:24:14.538133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.538108 5951 scope.go:122] "RemoveContainer" containerID="feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913" Nov 03 08:24:14.538690 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:14.538574 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913\": container with ID starting with feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913 not found: ID does not exist" containerID="feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913" Nov 03 08:24:14.538690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.538617 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913"} err="failed to get container status \"feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913\": rpc error: code = NotFound desc = could not find container \"feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913\": container with ID starting with feda5b48e07ea180667f7a1b54cf0ec465e87cd1875d46729796fe084351d913 not found: ID does not exist" Nov 03 08:24:14.542047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.542023 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b559e-bdbd55dbc-k8mp4"] Nov 03 08:24:14.552662 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.552610 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp"] Nov 03 08:24:14.556520 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.556497 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b559e-59f7698b6c-kq4qp"] Nov 03 08:24:14.737997 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.737963 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" path="/var/lib/kubelet/pods/5bb7caeb-48b0-4683-9dc2-08d96a124b64/volumes" Nov 03 08:24:14.738413 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.738399 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985bd2be-13e7-40a7-9022-38979241bc88" path="/var/lib/kubelet/pods/985bd2be-13e7-40a7-9022-38979241bc88/volumes" Nov 03 08:24:14.738805 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:14.738792 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" path="/var/lib/kubelet/pods/e9415064-28a2-41a2-bfb8-0a06634ab720/volumes" Nov 03 08:24:15.573831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.573163 5951 generic.go:356] "Generic (PLEG): container finished" podID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerID="bc9698e21c4b95531d254f9d04c75fa609d59dfa93d1251bbe743b4a3ce9b0cc" exitCode=0 Nov 03 08:24:15.573831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.573322 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" event={"ID":"545ab5b3-977a-4507-909e-6fbaedc3877f","Type":"ContainerDied","Data":"bc9698e21c4b95531d254f9d04c75fa609d59dfa93d1251bbe743b4a3ce9b0cc"} Nov 03 08:24:15.584097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.577555 5951 generic.go:356] "Generic (PLEG): container finished" podID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerID="ff45e8208f0978021b9b2563eef9723934a2a35e727e1b9d3dec5c6c8d4b6be1" exitCode=0 Nov 03 08:24:15.584097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.577657 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" event={"ID":"dd2df007-fdd1-4caa-b7e7-33477c96e02b","Type":"ContainerDied","Data":"ff45e8208f0978021b9b2563eef9723934a2a35e727e1b9d3dec5c6c8d4b6be1"} Nov 03 08:24:15.584097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.582149 5951 generic.go:356] "Generic (PLEG): container finished" podID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerID="e8a1dc870c69bdcdbde7a5642d218c9c64ff9d23b9f909b22e8958fbe9409252" exitCode=0 Nov 03 08:24:15.584097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.582341 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" event={"ID":"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7","Type":"ContainerDied","Data":"e8a1dc870c69bdcdbde7a5642d218c9c64ff9d23b9f909b22e8958fbe9409252"} Nov 03 08:24:15.605025 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.604990 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703339 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703396 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703471 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703508 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703540 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703665 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703742 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703793 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.703862 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token\") pod \"545ab5b3-977a-4507-909e-6fbaedc3877f\" (UID: \"545ab5b3-977a-4507-909e-6fbaedc3877f\") " Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.707057 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.707260 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.707944 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.708050 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.708750 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.708181 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.725908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.718994 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:15.725908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.721978 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.725908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.722009 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:15.725908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.722419 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r" pod "545ab5b3-977a-4507-909e-6fbaedc3877f" (UID: "545ab5b3-977a-4507-909e-6fbaedc3877f"). InnerVolumeSpecName "kube-api-access-mll7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:15.782336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.782291 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:24:15.787867 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.787823 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.805886 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.805970 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806073 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806165 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806211 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806213 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806250 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806277 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806299 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806320 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806368 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806422 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806449 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806492 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806557 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806611 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806672 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806742 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data\") pod \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\" (UID: \"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806803 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806823 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.806837 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28\") pod \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\" (UID: \"dd2df007-fdd1-4caa-b7e7-33477c96e02b\") " Nov 03 08:24:15.807709 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.807397 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808418 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808452 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808492 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808513 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808533 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mll7r\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-kube-api-access-mll7r\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808551 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808591 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808610 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545ab5b3-977a-4507-909e-6fbaedc3877f-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808628 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808662 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808678 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808694 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545ab5b3-977a-4507-909e-6fbaedc3877f-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.809215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.808717 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.811410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.809560 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.811410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.810163 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.811410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.810950 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.811410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.811171 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.812196 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.812169 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:15.813024 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.812689 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.819090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.819026 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:15.827739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.826319 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:15.827739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.826501 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:15.827739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.826810 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.833103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.833005 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "kube-api-access-4hz28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:15.836344 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.836319 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "kube-api-access-nk6rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:15.836450 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.836418 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy" pod "dd2df007-fdd1-4caa-b7e7-33477c96e02b" (UID: "dd2df007-fdd1-4caa-b7e7-33477c96e02b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:15.841224 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.841184 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo" pod "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" (UID: "2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:15.910487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910446 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910482 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910487 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910493 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6rs\" (UniqueName: \"kubernetes.io/projected/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-kube-api-access-nk6rs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910505 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910515 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910524 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910535 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910544 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910553 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hz28\" (UniqueName: \"kubernetes.io/projected/dd2df007-fdd1-4caa-b7e7-33477c96e02b-kube-api-access-4hz28\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910564 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910574 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910583 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910592 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:15.910918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:15.910602 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2df007-fdd1-4caa-b7e7-33477c96e02b-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:16.595093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.595058 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" Nov 03 08:24:16.595093 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.595086 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl" event={"ID":"545ab5b3-977a-4507-909e-6fbaedc3877f","Type":"ContainerDied","Data":"fc55b35b43dafa1f43865b044139113e14492b39fdd3a12a9d60f090b90f9bab"} Nov 03 08:24:16.595335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.595148 5951 scope.go:122] "RemoveContainer" containerID="bc9698e21c4b95531d254f9d04c75fa609d59dfa93d1251bbe743b4a3ce9b0cc" Nov 03 08:24:16.596962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.596925 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" event={"ID":"dd2df007-fdd1-4caa-b7e7-33477c96e02b","Type":"ContainerDied","Data":"c61e68f144c6cbfa431ad409699845df28566484b3f5653bea794654f96700a3"} Nov 03 08:24:16.597079 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.596926 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw" Nov 03 08:24:16.598955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.598928 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" event={"ID":"2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7","Type":"ContainerDied","Data":"995a5da07c83e79a3c8adf8d5034c6b352b41797881d47ee0603c1bf0187dcc2"} Nov 03 08:24:16.598955 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.598949 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84" Nov 03 08:24:16.614976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.614801 5951 scope.go:122] "RemoveContainer" containerID="ff45e8208f0978021b9b2563eef9723934a2a35e727e1b9d3dec5c6c8d4b6be1" Nov 03 08:24:16.631257 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.631230 5951 scope.go:122] "RemoveContainer" containerID="e8a1dc870c69bdcdbde7a5642d218c9c64ff9d23b9f909b22e8958fbe9409252" Nov 03 08:24:16.632134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.632110 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl"] Nov 03 08:24:16.635989 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.635963 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-q5bjl"] Nov 03 08:24:16.644159 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.644136 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw"] Nov 03 08:24:16.649938 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.649915 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8g6bw"] Nov 03 08:24:16.691293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.691251 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84"] Nov 03 08:24:16.697150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.697124 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-qtn84"] Nov 03 08:24:16.742376 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.742338 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" path="/var/lib/kubelet/pods/2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7/volumes" Nov 03 08:24:16.742952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.742935 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" path="/var/lib/kubelet/pods/545ab5b3-977a-4507-909e-6fbaedc3877f/volumes" Nov 03 08:24:16.743424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:16.743410 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" path="/var/lib/kubelet/pods/dd2df007-fdd1-4caa-b7e7-33477c96e02b/volumes" Nov 03 08:24:18.458761 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:18.458716 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" containerName="payload-pre-processing" containerID="cri-o://024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c" gracePeriod=25 Nov 03 08:24:19.029920 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.029700 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.062881 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.062978 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063254 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063288 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063323 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063365 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063479 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063503 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.064665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.063552 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token\") pod \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\" (UID: \"f2606b03-3902-4ea7-9eed-0dd6c35176e6\") " Nov 03 08:24:19.065266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.064911 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.065266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.065068 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.065443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.065415 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.066316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.066127 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.066316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.066288 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:19.068689 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.068664 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:19.068931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.068904 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:19.071398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.071327 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "kube-api-access-fssnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:19.071776 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.071752 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy" pod "f2606b03-3902-4ea7-9eed-0dd6c35176e6" (UID: "f2606b03-3902-4ea7-9eed-0dd6c35176e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.120619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.116362 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:24:19.165437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165399 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.165626 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165501 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.165626 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165552 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.165782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165627 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.165782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165761 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.165904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165778 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.165904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165808 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.165904 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165862 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.166052 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.165931 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.166106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166047 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token\") pod \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\" (UID: \"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3\") " Nov 03 08:24:19.166222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166195 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.166827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166757 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:19.166827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166776 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.166827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166803 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.166827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166820 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166834 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166848 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166863 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166876 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166889 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166904 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssnv\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-kube-api-access-fssnv\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166919 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2606b03-3902-4ea7-9eed-0dd6c35176e6-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.166933 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2606b03-3902-4ea7-9eed-0dd6c35176e6-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.167671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.167616 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.168058 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.168038 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.170106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.170077 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:19.170106 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.170095 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:19.170222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.170115 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "kube-api-access-zst52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:19.170222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.170116 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo" pod "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" (UID: "fa862535-d1bb-41ad-88d0-aa1fe91b4fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:19.268429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268334 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.268429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268371 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.268429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268384 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.268429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268397 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.268429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268409 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.268429 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268423 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zst52\" (UniqueName: \"kubernetes.io/projected/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-kube-api-access-zst52\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.268751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.268437 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:19.638303 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.638270 5951 generic.go:356] "Generic (PLEG): container finished" podID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerID="9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8" exitCode=0 Nov 03 08:24:19.638765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.638332 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" Nov 03 08:24:19.638765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.638368 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" event={"ID":"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3","Type":"ContainerDied","Data":"9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8"} Nov 03 08:24:19.638765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.638431 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8" event={"ID":"fa862535-d1bb-41ad-88d0-aa1fe91b4fa3","Type":"ContainerDied","Data":"2aeb12c0735bd0b4f870a331c6b62775033b6e4f4591e698332ff082f7f6801c"} Nov 03 08:24:19.638765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.638462 5951 scope.go:122] "RemoveContainer" containerID="9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8" Nov 03 08:24:19.640686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.640623 5951 generic.go:356] "Generic (PLEG): container finished" podID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerID="fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5" exitCode=0 Nov 03 08:24:19.640803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.640719 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" Nov 03 08:24:19.640883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.640742 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" event={"ID":"f2606b03-3902-4ea7-9eed-0dd6c35176e6","Type":"ContainerDied","Data":"fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5"} Nov 03 08:24:19.640943 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.640908 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk" event={"ID":"f2606b03-3902-4ea7-9eed-0dd6c35176e6","Type":"ContainerDied","Data":"52e125b04fd0a90577fa5d62eeef3b70c51fff6c6baeba03688bc13b0934d97d"} Nov 03 08:24:19.650638 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.650614 5951 scope.go:122] "RemoveContainer" containerID="9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8" Nov 03 08:24:19.651007 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:19.650978 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8\": container with ID starting with 9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8 not found: ID does not exist" containerID="9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8" Nov 03 08:24:19.651069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.651021 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8"} err="failed to get container status \"9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8\": rpc error: code = NotFound desc = could not find container \"9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8\": container with ID starting with 9bb7023e55f09ec72f3c4ef1114f7cd5553c6ea848412915b7eba8457a3d16a8 not found: ID does not exist" Nov 03 08:24:19.651069 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.651048 5951 scope.go:122] "RemoveContainer" containerID="fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5" Nov 03 08:24:19.661692 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.661626 5951 scope.go:122] "RemoveContainer" containerID="8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a" Nov 03 08:24:19.668874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.668817 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8"] Nov 03 08:24:19.672371 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.672348 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-k6kb8"] Nov 03 08:24:19.673827 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.673805 5951 scope.go:122] "RemoveContainer" containerID="fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5" Nov 03 08:24:19.674112 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:19.674093 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5\": container with ID starting with fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5 not found: ID does not exist" containerID="fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5" Nov 03 08:24:19.674157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.674121 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5"} err="failed to get container status \"fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5\": rpc error: code = NotFound desc = could not find container \"fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5\": container with ID starting with fd01bd50e46efb8dc56da523ce150c4717e36b4c4ff7cf64f9da9b30a84628a5 not found: ID does not exist" Nov 03 08:24:19.674157 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.674137 5951 scope.go:122] "RemoveContainer" containerID="8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a" Nov 03 08:24:19.674396 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:19.674381 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a\": container with ID starting with 8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a not found: ID does not exist" containerID="8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a" Nov 03 08:24:19.674442 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.674401 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a"} err="failed to get container status \"8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a\": rpc error: code = NotFound desc = could not find container \"8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a\": container with ID starting with 8fe4da29a72b2904311731d563061cf4d6f2488997c00721e5437f2ffb91899a not found: ID does not exist" Nov 03 08:24:19.680611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.680588 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk"] Nov 03 08:24:19.686621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:19.686600 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-rh7lk"] Nov 03 08:24:20.738706 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:20.738630 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" path="/var/lib/kubelet/pods/f2606b03-3902-4ea7-9eed-0dd6c35176e6/volumes" Nov 03 08:24:20.740486 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:20.740458 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" path="/var/lib/kubelet/pods/fa862535-d1bb-41ad-88d0-aa1fe91b4fa3/volumes" Nov 03 08:24:20.895519 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:20.895322 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892"] Nov 03 08:24:20.913697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:20.913581 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr"] Nov 03 08:24:21.799286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:21.799242 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:24:21.802859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:21.802835 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:24:22.480100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:22.477896 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2"] Nov 03 08:24:22.480100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:22.478155 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" containerID="cri-o://0e08cc96ea919fa77fa41dd14c77d5a9a4bba2d73223b63933586dca6bce25d1" gracePeriod=30 Nov 03 08:24:22.485555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:22.485529 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz"] Nov 03 08:24:22.486309 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:22.485772 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" containerID="cri-o://f8ed48c908418555fbe93a8735346e273ef90bfa0e65d1ae60a9308247817377" gracePeriod=30 Nov 03 08:24:22.498697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:22.496343 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv"] Nov 03 08:24:22.498697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:22.496530 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" containerID="cri-o://45405e2a4b8427150c4849c5ee4084c807a9adcf6c499951aa5ccc14f6a6d1dd" gracePeriod=30 Nov 03 08:24:24.096229 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:24.096183 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:25.895977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:25.895898 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" podUID="710222a7-fae0-41ae-be1e-0312f29fd512" containerName="payload-processing" containerID="cri-o://05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac" gracePeriod=25 Nov 03 08:24:25.914576 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:25.914503 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerName="payload-pre-processing" containerID="cri-o://50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d" gracePeriod=25 Nov 03 08:24:26.208237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.208197 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:24:26.296361 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.296307 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume\") pod \"710222a7-fae0-41ae-be1e-0312f29fd512\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " Nov 03 08:24:26.296571 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.296549 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964\" (UniqueName: \"kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964\") pod \"710222a7-fae0-41ae-be1e-0312f29fd512\" (UID: \"710222a7-fae0-41ae-be1e-0312f29fd512\") " Nov 03 08:24:26.297122 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.297094 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume" pod "710222a7-fae0-41ae-be1e-0312f29fd512" (UID: "710222a7-fae0-41ae-be1e-0312f29fd512"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:26.297325 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.297265 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/710222a7-fae0-41ae-be1e-0312f29fd512-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:26.299809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.299608 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964" pod "710222a7-fae0-41ae-be1e-0312f29fd512" (UID: "710222a7-fae0-41ae-be1e-0312f29fd512"). InnerVolumeSpecName "kube-api-access-zw964". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:26.398440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.398397 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw964\" (UniqueName: \"kubernetes.io/projected/710222a7-fae0-41ae-be1e-0312f29fd512-kube-api-access-zw964\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:26.759826 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.759793 5951 generic.go:356] "Generic (PLEG): container finished" podID="710222a7-fae0-41ae-be1e-0312f29fd512" containerID="05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac" exitCode=0 Nov 03 08:24:26.759985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.759846 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" Nov 03 08:24:26.759985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.759873 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" event={"ID":"710222a7-fae0-41ae-be1e-0312f29fd512","Type":"ContainerDied","Data":"05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac"} Nov 03 08:24:26.759985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.759929 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892" event={"ID":"710222a7-fae0-41ae-be1e-0312f29fd512","Type":"ContainerDied","Data":"dbba44adc1de97efacbce0a9994ab852b78b96f0f28d2b79dad356ec5926afed"} Nov 03 08:24:26.759985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.759950 5951 scope.go:122] "RemoveContainer" containerID="05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac" Nov 03 08:24:26.775633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.775609 5951 scope.go:122] "RemoveContainer" containerID="05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac" Nov 03 08:24:26.776434 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:26.776406 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac\": container with ID starting with 05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac not found: ID does not exist" containerID="05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac" Nov 03 08:24:26.776521 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.776441 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac"} err="failed to get container status \"05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac\": rpc error: code = NotFound desc = could not find container \"05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac\": container with ID starting with 05436cede725a4c1d9bf75a990602dd65f469f5ed4c384e6b49bf57d439b62ac not found: ID does not exist" Nov 03 08:24:26.778347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.778328 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892"] Nov 03 08:24:26.781572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:26.781550 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b8a152-76d844d85-rp892"] Nov 03 08:24:27.768612 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.768519 5951 generic.go:356] "Generic (PLEG): container finished" podID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerID="f8ed48c908418555fbe93a8735346e273ef90bfa0e65d1ae60a9308247817377" exitCode=0 Nov 03 08:24:27.768612 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.768593 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" event={"ID":"7b8bdfd9-02f4-4661-b758-67cba3c395e0","Type":"ContainerDied","Data":"f8ed48c908418555fbe93a8735346e273ef90bfa0e65d1ae60a9308247817377"} Nov 03 08:24:27.771299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.771269 5951 generic.go:356] "Generic (PLEG): container finished" podID="0f909765-4f94-4242-bdda-9f9d8f345397" containerID="0e08cc96ea919fa77fa41dd14c77d5a9a4bba2d73223b63933586dca6bce25d1" exitCode=0 Nov 03 08:24:27.771452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.771349 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" event={"ID":"0f909765-4f94-4242-bdda-9f9d8f345397","Type":"ContainerDied","Data":"0e08cc96ea919fa77fa41dd14c77d5a9a4bba2d73223b63933586dca6bce25d1"} Nov 03 08:24:27.774168 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.774141 5951 generic.go:356] "Generic (PLEG): container finished" podID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerID="45405e2a4b8427150c4849c5ee4084c807a9adcf6c499951aa5ccc14f6a6d1dd" exitCode=0 Nov 03 08:24:27.774295 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.774179 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" event={"ID":"77624484-a1cd-47bc-a1f0-9c132a0da250","Type":"ContainerDied","Data":"45405e2a4b8427150c4849c5ee4084c807a9adcf6c499951aa5ccc14f6a6d1dd"} Nov 03 08:24:27.848326 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.848291 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:24:27.853220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.853193 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:24:27.918015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.917980 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4"] Nov 03 08:24:27.930178 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.930150 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g"] Nov 03 08:24:27.937226 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:27.937198 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:24:28.041440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041389 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.041622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041450 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.041622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041488 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.041622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041525 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.041622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041556 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.041622 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041587 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.041931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041625 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.041931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041680 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.041931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041714 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.041931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041758 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.041931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041822 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.041931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041873 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.042240 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.041954 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.042292 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042257 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:28.042384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042356 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.042443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042380 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.042834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042745 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.042834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042777 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:28.042834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042793 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:28.042834 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042800 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042852 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042890 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042924 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.042999 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043095 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043144 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043184 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043236 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043287 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043328 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043356 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo\") pod \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\" (UID: \"7b8bdfd9-02f4-4661-b758-67cba3c395e0\") " Nov 03 08:24:28.043410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043397 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket\") pod \"77624484-a1cd-47bc-a1f0-9c132a0da250\" (UID: \"77624484-a1cd-47bc-a1f0-9c132a0da250\") " Nov 03 08:24:28.044302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043439 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.044302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043502 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.044302 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.043634 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token\") pod \"0f909765-4f94-4242-bdda-9f9d8f345397\" (UID: \"0f909765-4f94-4242-bdda-9f9d8f345397\") " Nov 03 08:24:28.044628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.044325 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.044628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.044501 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.044774 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.044736 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.045473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.045037 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.045473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.045220 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.045473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.045403 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.045473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.045421 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.046571 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046552 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046694 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046718 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046716 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046730 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046743 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046757 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046770 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046784 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f909765-4f94-4242-bdda-9f9d8f345397-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046798 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046812 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046826 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.046855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046841 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77624484-a1cd-47bc-a1f0-9c132a0da250-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.047550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046856 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.047550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046887 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.047550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.046901 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.047550 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.047266 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:28.048070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048050 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:28.048723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048552 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:24:28.048723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048584 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:28.048723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048623 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "kube-api-access-m69dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:28.048723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048627 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "kube-api-access-bz5k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:28.048723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048687 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy" pod "7b8bdfd9-02f4-4661-b758-67cba3c395e0" (UID: "7b8bdfd9-02f4-4661-b758-67cba3c395e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.048723 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048696 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.049254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.048885 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:28.049254 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.049070 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy" pod "77624484-a1cd-47bc-a1f0-9c132a0da250" (UID: "77624484-a1cd-47bc-a1f0-9c132a0da250"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:28.049625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.049606 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "kube-api-access-dgrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:28.050249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.050230 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token" pod "0f909765-4f94-4242-bdda-9f9d8f345397" (UID: "0f909765-4f94-4242-bdda-9f9d8f345397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148117 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f909765-4f94-4242-bdda-9f9d8f345397-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148155 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148171 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148188 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148203 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148218 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m69dw\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-kube-api-access-m69dw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148235 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrss\" (UniqueName: \"kubernetes.io/projected/0f909765-4f94-4242-bdda-9f9d8f345397-kube-api-access-dgrss\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148250 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148264 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5k7\" (UniqueName: \"kubernetes.io/projected/77624484-a1cd-47bc-a1f0-9c132a0da250-kube-api-access-bz5k7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148278 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f909765-4f94-4242-bdda-9f9d8f345397-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148294 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8bdfd9-02f4-4661-b758-67cba3c395e0-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.148338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.148308 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77624484-a1cd-47bc-a1f0-9c132a0da250-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:28.741032 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.740997 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710222a7-fae0-41ae-be1e-0312f29fd512" path="/var/lib/kubelet/pods/710222a7-fae0-41ae-be1e-0312f29fd512/volumes" Nov 03 08:24:28.782513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.782486 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" Nov 03 08:24:28.783037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.782509 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2" event={"ID":"0f909765-4f94-4242-bdda-9f9d8f345397","Type":"ContainerDied","Data":"4e8373b92a5e302296ddf18767c0403f973ad41008b2e71cdb0b8bfd72a18fbf"} Nov 03 08:24:28.783037 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.782605 5951 scope.go:122] "RemoveContainer" containerID="0e08cc96ea919fa77fa41dd14c77d5a9a4bba2d73223b63933586dca6bce25d1" Nov 03 08:24:28.784452 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.784436 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" Nov 03 08:24:28.784512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.784458 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv" event={"ID":"77624484-a1cd-47bc-a1f0-9c132a0da250","Type":"ContainerDied","Data":"742de3f7d1096e9ddc3ff675f94ee862db331f0f7bb1878f6c8c85babf3c4637"} Nov 03 08:24:28.786410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.786395 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" Nov 03 08:24:28.786508 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.786394 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz" event={"ID":"7b8bdfd9-02f4-4661-b758-67cba3c395e0","Type":"ContainerDied","Data":"2b57510486c785b941e953b026b70202f0b727fa1cc721767031f89dabaef544"} Nov 03 08:24:28.797068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.797049 5951 scope.go:122] "RemoveContainer" containerID="45405e2a4b8427150c4849c5ee4084c807a9adcf6c499951aa5ccc14f6a6d1dd" Nov 03 08:24:28.816785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.816750 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2"] Nov 03 08:24:28.816942 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.816795 5951 scope.go:122] "RemoveContainer" containerID="f8ed48c908418555fbe93a8735346e273ef90bfa0e65d1ae60a9308247817377" Nov 03 08:24:28.820318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.820291 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rvcj2"] Nov 03 08:24:28.827318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.827281 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv"] Nov 03 08:24:28.832439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.832412 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-cw6tv"] Nov 03 08:24:28.847481 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.847450 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz"] Nov 03 08:24:28.852416 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:28.852394 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-w95cz"] Nov 03 08:24:30.739728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:30.739595 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" path="/var/lib/kubelet/pods/0f909765-4f94-4242-bdda-9f9d8f345397/volumes" Nov 03 08:24:30.740260 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:30.740240 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" path="/var/lib/kubelet/pods/77624484-a1cd-47bc-a1f0-9c132a0da250/volumes" Nov 03 08:24:30.740782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:30.740761 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" path="/var/lib/kubelet/pods/7b8bdfd9-02f4-4661-b758-67cba3c395e0/volumes" Nov 03 08:24:31.962217 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:31.962180 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:24:32.055266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.055229 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr"] Nov 03 08:24:32.055527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.055490 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" podUID="533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" containerName="maas-api" containerID="cri-o://35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f" gracePeriod=30 Nov 03 08:24:32.590559 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.590528 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:24:32.676002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.675862 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn\" (UniqueName: \"kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn\") pod \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " Nov 03 08:24:32.676221 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.675976 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls\") pod \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " Nov 03 08:24:32.676221 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.676103 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs\") pod \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " Nov 03 08:24:32.676221 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.676138 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca\") pod \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\" (UID: \"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6\") " Nov 03 08:24:32.677174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.677147 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca" pod "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" (UID: "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:32.680821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.678580 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls" pod "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" (UID: "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:24:32.682161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.682137 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs" pod "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" (UID: "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:24:32.682294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.682243 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn" pod "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" (UID: "533b1d8c-6f35-4f7c-8e09-13a26bd34ff6"). InnerVolumeSpecName "kube-api-access-zsqbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:32.778686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.778614 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsqbn\" (UniqueName: \"kubernetes.io/projected/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-kube-api-access-zsqbn\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:32.778686 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.778669 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:32.778961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.778691 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:32.778961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.778714 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:32.785136 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.785101 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:32.843010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.842922 5951 generic.go:356] "Generic (PLEG): container finished" podID="533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" containerID="35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f" exitCode=0 Nov 03 08:24:32.843010 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.842969 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" Nov 03 08:24:32.843222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.843005 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" event={"ID":"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6","Type":"ContainerDied","Data":"35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f"} Nov 03 08:24:32.843222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.843061 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr" event={"ID":"533b1d8c-6f35-4f7c-8e09-13a26bd34ff6","Type":"ContainerDied","Data":"2d199dd1e2776ee325e7c18a0d9d1e161dc0e3ea68ab8833ed07793978a1e22f"} Nov 03 08:24:32.843222 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.843087 5951 scope.go:122] "RemoveContainer" containerID="35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f" Nov 03 08:24:32.853766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.853735 5951 scope.go:122] "RemoveContainer" containerID="35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f" Nov 03 08:24:32.854052 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:32.854026 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f\": container with ID starting with 35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f not found: ID does not exist" containerID="35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f" Nov 03 08:24:32.854139 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.854052 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f"} err="failed to get container status \"35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f\": rpc error: code = NotFound desc = could not find container \"35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f\": container with ID starting with 35977b0f08b228f29fb57f9cf883997f7e8275ded268331e3f545922082bdf2f not found: ID does not exist" Nov 03 08:24:32.860002 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.859980 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr"] Nov 03 08:24:32.864778 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:32.864753 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b8a152-599648c95b-4skrr"] Nov 03 08:24:33.327411 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:33.327370 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj"] Nov 03 08:24:34.124987 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:34.124938 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:34.738253 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:34.738220 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" path="/var/lib/kubelet/pods/533b1d8c-6f35-4f7c-8e09-13a26bd34ff6/volumes" Nov 03 08:24:38.328500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.328457 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" podUID="75476f61-7722-4160-b920-18d413090624" containerName="payload-pre-processing" containerID="cri-o://2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645" gracePeriod=25 Nov 03 08:24:38.777792 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.777763 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:24:38.909667 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.909548 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4\" (UniqueName: \"kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4\") pod \"a9a65b48-df70-4abb-bf90-32d68094d841\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " Nov 03 08:24:38.910787 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.910761 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume\") pod \"a9a65b48-df70-4abb-bf90-32d68094d841\" (UID: \"a9a65b48-df70-4abb-bf90-32d68094d841\") " Nov 03 08:24:38.913945 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.913915 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume" pod "a9a65b48-df70-4abb-bf90-32d68094d841" (UID: "a9a65b48-df70-4abb-bf90-32d68094d841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:38.926577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.926528 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4" pod "a9a65b48-df70-4abb-bf90-32d68094d841" (UID: "a9a65b48-df70-4abb-bf90-32d68094d841"). InnerVolumeSpecName "kube-api-access-cgbb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:38.927282 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.927252 5951 generic.go:356] "Generic (PLEG): container finished" podID="a9a65b48-df70-4abb-bf90-32d68094d841" containerID="024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c" exitCode=0 Nov 03 08:24:38.927618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.927601 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" event={"ID":"a9a65b48-df70-4abb-bf90-32d68094d841","Type":"ContainerDied","Data":"024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c"} Nov 03 08:24:38.927818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.927803 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" event={"ID":"a9a65b48-df70-4abb-bf90-32d68094d841","Type":"ContainerDied","Data":"4582db3709205f072eb54d26b5c0890173d723a5ec6eba06f5a4d8a6c181af43"} Nov 03 08:24:38.927933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.927922 5951 scope.go:122] "RemoveContainer" containerID="024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c" Nov 03 08:24:38.928175 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.928160 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq" Nov 03 08:24:38.953840 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.953813 5951 scope.go:122] "RemoveContainer" containerID="024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c" Nov 03 08:24:38.955964 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:38.955926 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c\": container with ID starting with 024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c not found: ID does not exist" containerID="024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c" Nov 03 08:24:38.956191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.956155 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c"} err="failed to get container status \"024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c\": rpc error: code = NotFound desc = could not find container \"024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c\": container with ID starting with 024f1083ac4fcf75a6d16c2c7e9445337e281312fd0ca9dcaf15f77a2defa41c not found: ID does not exist" Nov 03 08:24:38.958847 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.958824 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq"] Nov 03 08:24:38.965549 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:38.965523 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-616e46-6f957d44c5-cd5dq"] Nov 03 08:24:39.013713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:39.013670 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgbb4\" (UniqueName: \"kubernetes.io/projected/a9a65b48-df70-4abb-bf90-32d68094d841-kube-api-access-cgbb4\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:39.013713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:39.013707 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a65b48-df70-4abb-bf90-32d68094d841-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:40.743794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:40.743763 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" path="/var/lib/kubelet/pods/a9a65b48-df70-4abb-bf90-32d68094d841/volumes" Nov 03 08:24:41.955169 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:41.955134 5951 generic.go:356] "Generic (PLEG): container finished" podID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerID="3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad" exitCode=0 Nov 03 08:24:41.955773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:41.955211 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" event={"ID":"3c6aa698-50f9-4827-bfb2-3af5344613cd","Type":"ContainerDied","Data":"3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad"} Nov 03 08:24:41.955773 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:41.955753 5951 scope.go:122] "RemoveContainer" containerID="3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad" Nov 03 08:24:41.957296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:41.957266 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:24:41.957384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:41.957339 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:24:41.957384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:41.957370 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:24:42.785107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:42.785071 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:42.953012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:42.952940 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" containerID="cri-o://4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292" gracePeriod=45 Nov 03 08:24:42.965167 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:42.965137 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" event={"ID":"3c6aa698-50f9-4827-bfb2-3af5344613cd","Type":"ContainerStarted","Data":"3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84"} Nov 03 08:24:42.966803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:42.966777 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" containerID="cri-o://0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf" gracePeriod=45 Nov 03 08:24:43.240354 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.239570 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused" Nov 03 08:24:43.361631 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.361599 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:24:43.449036 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.449009 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:24:43.473747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.473718 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:24:43.474777 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.474744 5951 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 08:24:43.474920 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.474811 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 08:24:43.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.517767 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm\") pod \"3da169da-7303-4040-85c6-19d02852faa8\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " Nov 03 08:24:43.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.517932 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9\" (UniqueName: \"kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9\") pod \"3da169da-7303-4040-85c6-19d02852faa8\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " Nov 03 08:24:43.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.518114 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home\") pod \"3da169da-7303-4040-85c6-19d02852faa8\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " Nov 03 08:24:43.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.518287 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir\") pod \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " Nov 03 08:24:43.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.518408 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache\") pod \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " Nov 03 08:24:43.523669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.521800 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir\") pod \"3da169da-7303-4040-85c6-19d02852faa8\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " Nov 03 08:24:43.535380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.532979 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir" pod "52a712fa-027b-4ece-81a9-a9a7c9c8c19e" (UID: "52a712fa-027b-4ece-81a9-a9a7c9c8c19e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.535380 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.533484 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home" pod "3da169da-7303-4040-85c6-19d02852faa8" (UID: "3da169da-7303-4040-85c6-19d02852faa8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.535618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535398 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir" pod "3da169da-7303-4040-85c6-19d02852faa8" (UID: "3da169da-7303-4040-85c6-19d02852faa8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.535618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535417 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache" pod "52a712fa-027b-4ece-81a9-a9a7c9c8c19e" (UID: "52a712fa-027b-4ece-81a9-a9a7c9c8c19e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.535618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535507 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home\") pod \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " Nov 03 08:24:43.535862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535693 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home" pod "52a712fa-027b-4ece-81a9-a9a7c9c8c19e" (UID: "52a712fa-027b-4ece-81a9-a9a7c9c8c19e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.535862 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535707 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache\") pod \"3da169da-7303-4040-85c6-19d02852faa8\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " Nov 03 08:24:43.535972 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535874 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache" pod "3da169da-7303-4040-85c6-19d02852faa8" (UID: "3da169da-7303-4040-85c6-19d02852faa8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.536029 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.535983 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs\" (UniqueName: \"kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs\") pod \"3da169da-7303-4040-85c6-19d02852faa8\" (UID: \"3da169da-7303-4040-85c6-19d02852faa8\") " Nov 03 08:24:43.536029 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.536016 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct\" (UniqueName: \"kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct\") pod \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " Nov 03 08:24:43.537294 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.537268 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9" pod "3da169da-7303-4040-85c6-19d02852faa8" (UID: "3da169da-7303-4040-85c6-19d02852faa8"). InnerVolumeSpecName "kube-api-access-t7wg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:43.537527 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.537503 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm\") pod \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " Nov 03 08:24:43.537908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.537614 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs\") pod \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\" (UID: \"52a712fa-027b-4ece-81a9-a9a7c9c8c19e\") " Nov 03 08:24:43.538458 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538434 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wg9\" (UniqueName: \"kubernetes.io/projected/3da169da-7303-4040-85c6-19d02852faa8-kube-api-access-t7wg9\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.538604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538463 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.538604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538478 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.538604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538489 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.538604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538503 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.538604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538519 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.538604 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.538533 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.539328 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.539296 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs" pod "3da169da-7303-4040-85c6-19d02852faa8" (UID: "3da169da-7303-4040-85c6-19d02852faa8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:24:43.539629 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.539602 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm" pod "3da169da-7303-4040-85c6-19d02852faa8" (UID: "3da169da-7303-4040-85c6-19d02852faa8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.540988 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.540944 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs" pod "52a712fa-027b-4ece-81a9-a9a7c9c8c19e" (UID: "52a712fa-027b-4ece-81a9-a9a7c9c8c19e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:24:43.541212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.541186 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct" pod "52a712fa-027b-4ece-81a9-a9a7c9c8c19e" (UID: "52a712fa-027b-4ece-81a9-a9a7c9c8c19e"). InnerVolumeSpecName "kube-api-access-kftct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:43.542046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.541991 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm" pod "52a712fa-027b-4ece-81a9-a9a7c9c8c19e" (UID: "52a712fa-027b-4ece-81a9-a9a7c9c8c19e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:24:43.641084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.639949 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da169da-7303-4040-85c6-19d02852faa8-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.641084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.639980 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftct\" (UniqueName: \"kubernetes.io/projected/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-kube-api-access-kftct\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.641084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.639990 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.641084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.639999 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a712fa-027b-4ece-81a9-a9a7c9c8c19e-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.641084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.640008 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da169da-7303-4040-85c6-19d02852faa8-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:43.976788 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.976729 5951 generic.go:356] "Generic (PLEG): container finished" podID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerID="4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292" exitCode=0 Nov 03 08:24:43.977225 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.976916 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" Nov 03 08:24:43.977660 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.977620 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" event={"ID":"52a712fa-027b-4ece-81a9-a9a7c9c8c19e","Type":"ContainerDied","Data":"4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292"} Nov 03 08:24:43.977764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.977684 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4" event={"ID":"52a712fa-027b-4ece-81a9-a9a7c9c8c19e","Type":"ContainerDied","Data":"2f25da349b03bd7f9906cd61ae9b509c5b792006a6bf769fb0b1e6abfbcab4fc"} Nov 03 08:24:43.977764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.977707 5951 scope.go:122] "RemoveContainer" containerID="4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292" Nov 03 08:24:43.979721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.979689 5951 generic.go:356] "Generic (PLEG): container finished" podID="3da169da-7303-4040-85c6-19d02852faa8" containerID="0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf" exitCode=0 Nov 03 08:24:43.980220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.979734 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" Nov 03 08:24:43.980220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.979808 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" event={"ID":"3da169da-7303-4040-85c6-19d02852faa8","Type":"ContainerDied","Data":"0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf"} Nov 03 08:24:43.980220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.979841 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" event={"ID":"3da169da-7303-4040-85c6-19d02852faa8","Type":"ContainerDied","Data":"5e1c4b231d9d0bcfd303978e5d31ab6fcab3a5a4435c8f0bf0c80f3b32c85497"} Nov 03 08:24:43.993531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:43.993511 5951 scope.go:122] "RemoveContainer" containerID="4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292" Nov 03 08:24:44.023301 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:44.021314 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292\": container with ID starting with 4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292 not found: ID does not exist" containerID="4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292" Nov 03 08:24:44.023301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.021366 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292"} err="failed to get container status \"4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292\": rpc error: code = NotFound desc = could not find container \"4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292\": container with ID starting with 4dbcee0d6bdd66a2408a0ee9a24b4ba6d6ec6c27deaa3d5e6bbdecf903b8e292 not found: ID does not exist" Nov 03 08:24:44.023301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.021392 5951 scope.go:122] "RemoveContainer" containerID="0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf" Nov 03 08:24:44.041514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.041403 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4"] Nov 03 08:24:44.041603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.041532 5951 scope.go:122] "RemoveContainer" containerID="0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf" Nov 03 08:24:44.041891 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:44.041868 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf\": container with ID starting with 0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf not found: ID does not exist" containerID="0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf" Nov 03 08:24:44.041990 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.041897 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf"} err="failed to get container status \"0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf\": rpc error: code = NotFound desc = could not find container \"0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf\": container with ID starting with 0fb6b1fa811cd2686049fa6989aecabf6cea71cc729e550e666f3b99b6a87ddf not found: ID does not exist" Nov 03 08:24:44.045179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.045156 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/distinct-2-w3-3e88b7-kserve-6877f7f558-ff5w4"] Nov 03 08:24:44.050390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.050358 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g"] Nov 03 08:24:44.053388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.053365 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g"] Nov 03 08:24:44.317783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.317734 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e88b7/premium-w3-3e88b7-kserve-76469f77f4-8km7g" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 08:24:44.474092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.474056 5951 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 08:24:44.474293 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.474131 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 08:24:44.675175 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.675069 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" podUID="75476f61-7722-4160-b920-18d413090624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:44.760264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.760221 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da169da-7303-4040-85c6-19d02852faa8" path="/var/lib/kubelet/pods/3da169da-7303-4040-85c6-19d02852faa8/volumes" Nov 03 08:24:44.761741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:44.760881 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" path="/var/lib/kubelet/pods/52a712fa-027b-4ece-81a9-a9a7c9c8c19e/volumes" Nov 03 08:24:45.474447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:45.474403 5951 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 08:24:45.474961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:45.474453 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 08:24:46.277278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.277253 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:24:46.277803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.277783 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume\") pod \"c3eff667-e577-4878-bf46-4fc8a264aef0\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " Nov 03 08:24:46.277910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.277897 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6\" (UniqueName: \"kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6\") pod \"c3eff667-e577-4878-bf46-4fc8a264aef0\" (UID: \"c3eff667-e577-4878-bf46-4fc8a264aef0\") " Nov 03 08:24:46.278128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.278109 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume" pod "c3eff667-e577-4878-bf46-4fc8a264aef0" (UID: "c3eff667-e577-4878-bf46-4fc8a264aef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:46.278682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.278662 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eff667-e577-4878-bf46-4fc8a264aef0-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:46.280116 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.280084 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6" pod "c3eff667-e577-4878-bf46-4fc8a264aef0" (UID: "c3eff667-e577-4878-bf46-4fc8a264aef0"). InnerVolumeSpecName "kube-api-access-b7qb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:46.379868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.379834 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qb6\" (UniqueName: \"kubernetes.io/projected/c3eff667-e577-4878-bf46-4fc8a264aef0-kube-api-access-b7qb6\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:46.474447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.474412 5951 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 08:24:46.474676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:46.474633 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 08:24:47.021916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.021821 5951 generic.go:356] "Generic (PLEG): container finished" podID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerID="50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d" exitCode=0 Nov 03 08:24:47.022179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.022159 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" event={"ID":"c3eff667-e577-4878-bf46-4fc8a264aef0","Type":"ContainerDied","Data":"50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d"} Nov 03 08:24:47.022360 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.022345 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" event={"ID":"c3eff667-e577-4878-bf46-4fc8a264aef0","Type":"ContainerDied","Data":"27007472d32f3a224f69e071d0829b6e05fc1a7432741d0b6eb50fff9d747a4e"} Nov 03 08:24:47.022479 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.022467 5951 scope.go:122] "RemoveContainer" containerID="50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d" Nov 03 08:24:47.028133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.028107 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr" Nov 03 08:24:47.043951 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.043778 5951 scope.go:122] "RemoveContainer" containerID="50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d" Nov 03 08:24:47.044177 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:47.044150 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d\": container with ID starting with 50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d not found: ID does not exist" containerID="50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d" Nov 03 08:24:47.044327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.044303 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d"} err="failed to get container status \"50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d\": rpc error: code = NotFound desc = could not find container \"50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d\": container with ID starting with 50181c5e36248e98119ecdb4137cf6462485731b541368f837f6252108524a8d not found: ID does not exist" Nov 03 08:24:47.050177 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.050150 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr"] Nov 03 08:24:47.056494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.056468 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b8a152-599c77585f-xfrvr"] Nov 03 08:24:47.474097 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.474058 5951 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 08:24:47.474273 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.474110 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 08:24:47.845341 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:47.845303 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:24:48.629197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:48.629154 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:24:48.638144 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:48.638111 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:24:48.742271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:48.742231 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" path="/var/lib/kubelet/pods/c3eff667-e577-4878-bf46-4fc8a264aef0/volumes" Nov 03 08:24:50.017415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:50.017337 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb"] Nov 03 08:24:50.042946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:50.042912 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6"] Nov 03 08:24:54.674946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:54.674912 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" podUID="75476f61-7722-4160-b920-18d413090624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:24:55.018589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:55.018480 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerName="payload-processing" containerID="cri-o://89d7a7844e2224149fca5e7a8a6473ae21b50bceabf7f15751641a830112566d" gracePeriod=25 Nov 03 08:24:55.043296 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:55.043258 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerName="payload-pre-processing" containerID="cri-o://8ba2726ae3299ec24374c8489192106b95b305a9e21f39d2201a368fa17aef01" gracePeriod=25 Nov 03 08:24:56.624535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:56.624175 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq"] Nov 03 08:24:56.624535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:56.624259 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr"] Nov 03 08:24:56.624535 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:56.624471 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" containerID="cri-o://56b02f9cf76de38df299646285c62cfb68c3953b0735da8374a10ad55fc28086" gracePeriod=30 Nov 03 08:24:56.625277 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:56.624769 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" containerID="cri-o://a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a" gracePeriod=30 Nov 03 08:24:58.621511 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.621479 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:24:58.678457 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.678412 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j\" (UniqueName: \"kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j\") pod \"75476f61-7722-4160-b920-18d413090624\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " Nov 03 08:24:58.678657 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.678599 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume\") pod \"75476f61-7722-4160-b920-18d413090624\" (UID: \"75476f61-7722-4160-b920-18d413090624\") " Nov 03 08:24:58.679084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.679054 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume" pod "75476f61-7722-4160-b920-18d413090624" (UID: "75476f61-7722-4160-b920-18d413090624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:24:58.680745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.680721 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j" pod "75476f61-7722-4160-b920-18d413090624" (UID: "75476f61-7722-4160-b920-18d413090624"). InnerVolumeSpecName "kube-api-access-dzw6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:24:58.780823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.780779 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzw6j\" (UniqueName: \"kubernetes.io/projected/75476f61-7722-4160-b920-18d413090624-kube-api-access-dzw6j\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:58.780823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:58.780811 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75476f61-7722-4160-b920-18d413090624-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:24:59.130089 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.130054 5951 generic.go:356] "Generic (PLEG): container finished" podID="75476f61-7722-4160-b920-18d413090624" containerID="2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645" exitCode=0 Nov 03 08:24:59.130298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.130102 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" Nov 03 08:24:59.130298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.130137 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" event={"ID":"75476f61-7722-4160-b920-18d413090624","Type":"ContainerDied","Data":"2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645"} Nov 03 08:24:59.130298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.130196 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj" event={"ID":"75476f61-7722-4160-b920-18d413090624","Type":"ContainerDied","Data":"034ae27e0ee14ef69cfba1eb0df6b266527adc5ccb20ffb2c9e764a6944dc3d9"} Nov 03 08:24:59.130502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.130481 5951 scope.go:122] "RemoveContainer" containerID="2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645" Nov 03 08:24:59.141735 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.141718 5951 scope.go:122] "RemoveContainer" containerID="2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645" Nov 03 08:24:59.142039 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:24:59.142020 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645\": container with ID starting with 2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645 not found: ID does not exist" containerID="2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645" Nov 03 08:24:59.142099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.142047 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645"} err="failed to get container status \"2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645\": rpc error: code = NotFound desc = could not find container \"2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645\": container with ID starting with 2a3f665acd056b02f660a4f6bc73c0c3e818c3fb1dd4c409aaf38f41dc842645 not found: ID does not exist" Nov 03 08:24:59.153675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.153611 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj"] Nov 03 08:24:59.165451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:24:59.165427 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e88b7-7dc44c97d7-4r2wj"] Nov 03 08:25:00.739530 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:00.739491 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75476f61-7722-4160-b920-18d413090624" path="/var/lib/kubelet/pods/75476f61-7722-4160-b920-18d413090624/volumes" Nov 03 08:25:01.942871 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:01.942319 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:25:01.970254 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:25:01.970214 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:25:02.050939 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.050903 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051119 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051052 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051180 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051125 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051249 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051178 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051256 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051308 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051290 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051409 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051574 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051579 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.051779 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051620 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy\") pod \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\" (UID: \"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4\") " Nov 03 08:25:02.051779 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051681 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.051879 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.051827 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.052272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.052237 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.052272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.052270 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.052427 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.052284 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.052837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.052764 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:25:02.053045 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.053022 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.054090 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.054061 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "kube-api-access-wlkvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:02.054712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.054686 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.055533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.055502 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:02.055935 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.055904 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo" pod "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" (UID: "aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:25:02.154597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.154519 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.154597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.154549 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.154597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.154560 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.154597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.154569 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.154597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.154579 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkvc\" (UniqueName: \"kubernetes.io/projected/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-kube-api-access-wlkvc\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.154597 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.154590 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.169032 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.169001 5951 generic.go:356] "Generic (PLEG): container finished" podID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerID="56b02f9cf76de38df299646285c62cfb68c3953b0735da8374a10ad55fc28086" exitCode=0 Nov 03 08:25:02.169189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.169051 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" event={"ID":"5b76e67a-2410-43d6-b272-2498ac21edc2","Type":"ContainerDied","Data":"56b02f9cf76de38df299646285c62cfb68c3953b0735da8374a10ad55fc28086"} Nov 03 08:25:02.169189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.169134 5951 scope.go:122] "RemoveContainer" containerID="55d7bb448c21d6e5f91180baafef91ed747fe273818d293b0ccb9f97f7bc1f00" Nov 03 08:25:02.170883 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.170859 5951 generic.go:356] "Generic (PLEG): container finished" podID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerID="a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a" exitCode=0 Nov 03 08:25:02.171003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.170912 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" Nov 03 08:25:02.171003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.170925 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" event={"ID":"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4","Type":"ContainerDied","Data":"a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a"} Nov 03 08:25:02.171003 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.170977 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq" event={"ID":"aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4","Type":"ContainerDied","Data":"537b6f0dc2c3e1394f2056872b84e21e97c2ec747468ff607f712cab6311e2e1"} Nov 03 08:25:02.183678 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.183620 5951 scope.go:122] "RemoveContainer" containerID="a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a" Nov 03 08:25:02.193504 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.193470 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq"] Nov 03 08:25:02.197625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.197539 5951 scope.go:122] "RemoveContainer" containerID="a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a" Nov 03 08:25:02.198078 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:25:02.198053 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a\": container with ID starting with a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a not found: ID does not exist" containerID="a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a" Nov 03 08:25:02.198167 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.198091 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a"} err="failed to get container status \"a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a\": rpc error: code = NotFound desc = could not find container \"a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a\": container with ID starting with a28fcd77770658ea33de249c3e36d2aafd990cac5a784462929aeb3d46ab132a not found: ID does not exist" Nov 03 08:25:02.200914 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.200880 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-ksrwq"] Nov 03 08:25:02.313569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.313541 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:25:02.458134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458035 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458079 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458134 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458126 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458192 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458343 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458382 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458446 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458397 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:25:02.458751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458496 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458616 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458660 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.458751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458696 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data\") pod \"5b76e67a-2410-43d6-b272-2498ac21edc2\" (UID: \"5b76e67a-2410-43d6-b272-2498ac21edc2\") " Nov 03 08:25:02.458751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458695 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.459206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458880 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.459206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.458971 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.459297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.459227 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.459297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.459242 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b76e67a-2410-43d6-b272-2498ac21edc2-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.459297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.459251 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.459297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.459260 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.459297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.459269 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.460583 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.460563 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:25:02.460927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.460907 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "kube-api-access-57xrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:02.461082 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.461070 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:02.461230 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.461216 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token" pod "5b76e67a-2410-43d6-b272-2498ac21edc2" (UID: "5b76e67a-2410-43d6-b272-2498ac21edc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:02.560795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.560757 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.560795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.560791 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57xrt\" (UniqueName: \"kubernetes.io/projected/5b76e67a-2410-43d6-b272-2498ac21edc2-kube-api-access-57xrt\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.560795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.560800 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.561056 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.560809 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b76e67a-2410-43d6-b272-2498ac21edc2-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:02.614683 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.614613 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:25:02.738118 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:02.738022 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" path="/var/lib/kubelet/pods/aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4/volumes" Nov 03 08:25:03.179917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:03.179888 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" Nov 03 08:25:03.180378 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:03.179913 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr" event={"ID":"5b76e67a-2410-43d6-b272-2498ac21edc2","Type":"ContainerDied","Data":"4a7870f3a993a3a83e2c61ee83f4b2b09062bbaf1a950250eddeaee05fdf2170"} Nov 03 08:25:03.180378 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:03.179971 5951 scope.go:122] "RemoveContainer" containerID="56b02f9cf76de38df299646285c62cfb68c3953b0735da8374a10ad55fc28086" Nov 03 08:25:03.199817 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:03.199776 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr"] Nov 03 08:25:03.203193 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:03.203165 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-mh6gr"] Nov 03 08:25:03.635658 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:03.635614 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:25:04.746893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:04.746856 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" path="/var/lib/kubelet/pods/5b76e67a-2410-43d6-b272-2498ac21edc2/volumes" Nov 03 08:25:10.725440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:10.721530 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2"] Nov 03 08:25:10.725440 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:10.721934 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" podUID="a1598327-6f2f-47c8-963e-4587fbf725cf" containerName="maas-api" containerID="cri-o://9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0" gracePeriod=30 Nov 03 08:25:11.140112 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.140083 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:25:11.258874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.258833 5951 generic.go:356] "Generic (PLEG): container finished" podID="a1598327-6f2f-47c8-963e-4587fbf725cf" containerID="9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0" exitCode=0 Nov 03 08:25:11.259043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.258888 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" Nov 03 08:25:11.259043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.258926 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" event={"ID":"a1598327-6f2f-47c8-963e-4587fbf725cf","Type":"ContainerDied","Data":"9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0"} Nov 03 08:25:11.259043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.258998 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2" event={"ID":"a1598327-6f2f-47c8-963e-4587fbf725cf","Type":"ContainerDied","Data":"51989cfcdbbda552de91becabc7b196c7cf68054447217d4c564dec1390cb9f7"} Nov 03 08:25:11.259043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.259031 5951 scope.go:122] "RemoveContainer" containerID="9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0" Nov 03 08:25:11.270882 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.270834 5951 scope.go:122] "RemoveContainer" containerID="9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0" Nov 03 08:25:11.271257 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:25:11.271230 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0\": container with ID starting with 9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0 not found: ID does not exist" containerID="9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0" Nov 03 08:25:11.271476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.271429 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0"} err="failed to get container status \"9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0\": rpc error: code = NotFound desc = could not find container \"9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0\": container with ID starting with 9c8cf695888ab6a6463078bca6c7b416b55576f01d4eb223c686c3d2d709f7c0 not found: ID does not exist" Nov 03 08:25:11.301832 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.301800 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca\") pod \"a1598327-6f2f-47c8-963e-4587fbf725cf\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " Nov 03 08:25:11.302017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.301882 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs\") pod \"a1598327-6f2f-47c8-963e-4587fbf725cf\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " Nov 03 08:25:11.302017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.301951 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls\") pod \"a1598327-6f2f-47c8-963e-4587fbf725cf\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " Nov 03 08:25:11.302017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.301974 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv\" (UniqueName: \"kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv\") pod \"a1598327-6f2f-47c8-963e-4587fbf725cf\" (UID: \"a1598327-6f2f-47c8-963e-4587fbf725cf\") " Nov 03 08:25:11.302262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.302239 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca" pod "a1598327-6f2f-47c8-963e-4587fbf725cf" (UID: "a1598327-6f2f-47c8-963e-4587fbf725cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:25:11.302503 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.302477 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1598327-6f2f-47c8-963e-4587fbf725cf-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:11.307263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.307223 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv" pod "a1598327-6f2f-47c8-963e-4587fbf725cf" (UID: "a1598327-6f2f-47c8-963e-4587fbf725cf"). InnerVolumeSpecName "kube-api-access-22mfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:11.307430 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.307285 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs" pod "a1598327-6f2f-47c8-963e-4587fbf725cf" (UID: "a1598327-6f2f-47c8-963e-4587fbf725cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:25:11.308149 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.308126 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls" pod "a1598327-6f2f-47c8-963e-4587fbf725cf" (UID: "a1598327-6f2f-47c8-963e-4587fbf725cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:25:11.403596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.403499 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:11.403596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.403532 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1598327-6f2f-47c8-963e-4587fbf725cf-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:11.403596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.403550 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22mfv\" (UniqueName: \"kubernetes.io/projected/a1598327-6f2f-47c8-963e-4587fbf725cf-kube-api-access-22mfv\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:11.582415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.582383 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2"] Nov 03 08:25:11.587023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:11.586991 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bd085d-5774ffc8f6-7m2v2"] Nov 03 08:25:12.614065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:12.614030 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:25:12.739959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:12.739923 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1598327-6f2f-47c8-963e-4587fbf725cf" path="/var/lib/kubelet/pods/a1598327-6f2f-47c8-963e-4587fbf725cf/volumes" Nov 03 08:25:13.636078 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:13.636038 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:25:15.343954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.343913 5951 generic.go:356] "Generic (PLEG): container finished" podID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerID="8ba2726ae3299ec24374c8489192106b95b305a9e21f39d2201a368fa17aef01" exitCode=0 Nov 03 08:25:15.344571 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.344256 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" event={"ID":"61ecfdea-2575-4a74-aeed-2a9068ae0d0f","Type":"ContainerDied","Data":"8ba2726ae3299ec24374c8489192106b95b305a9e21f39d2201a368fa17aef01"} Nov 03 08:25:15.346915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.346876 5951 generic.go:356] "Generic (PLEG): container finished" podID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerID="89d7a7844e2224149fca5e7a8a6473ae21b50bceabf7f15751641a830112566d" exitCode=0 Nov 03 08:25:15.347080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.347063 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" event={"ID":"aec249df-0e55-474e-ad96-e8a0769fd2bd","Type":"ContainerDied","Data":"89d7a7844e2224149fca5e7a8a6473ae21b50bceabf7f15751641a830112566d"} Nov 03 08:25:15.388304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.388273 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:25:15.391763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.391736 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:25:15.455582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.455550 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume\") pod \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " Nov 03 08:25:15.455782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.455631 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw\" (UniqueName: \"kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw\") pod \"aec249df-0e55-474e-ad96-e8a0769fd2bd\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " Nov 03 08:25:15.455782 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.455725 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47\" (UniqueName: \"kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47\") pod \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\" (UID: \"61ecfdea-2575-4a74-aeed-2a9068ae0d0f\") " Nov 03 08:25:15.455934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.455842 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume\") pod \"aec249df-0e55-474e-ad96-e8a0769fd2bd\" (UID: \"aec249df-0e55-474e-ad96-e8a0769fd2bd\") " Nov 03 08:25:15.456063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.456034 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume" pod "61ecfdea-2575-4a74-aeed-2a9068ae0d0f" (UID: "61ecfdea-2575-4a74-aeed-2a9068ae0d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:25:15.456272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.456250 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume" pod "aec249df-0e55-474e-ad96-e8a0769fd2bd" (UID: "aec249df-0e55-474e-ad96-e8a0769fd2bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:25:15.456399 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.456385 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec249df-0e55-474e-ad96-e8a0769fd2bd-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:15.456636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.456401 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:15.458117 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.458088 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47" pod "61ecfdea-2575-4a74-aeed-2a9068ae0d0f" (UID: "61ecfdea-2575-4a74-aeed-2a9068ae0d0f"). InnerVolumeSpecName "kube-api-access-snl47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:15.459061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.459037 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw" pod "aec249df-0e55-474e-ad96-e8a0769fd2bd" (UID: "aec249df-0e55-474e-ad96-e8a0769fd2bd"). InnerVolumeSpecName "kube-api-access-ppjrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:15.557839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.557806 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjrw\" (UniqueName: \"kubernetes.io/projected/aec249df-0e55-474e-ad96-e8a0769fd2bd-kube-api-access-ppjrw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:15.557839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:15.557835 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snl47\" (UniqueName: \"kubernetes.io/projected/61ecfdea-2575-4a74-aeed-2a9068ae0d0f-kube-api-access-snl47\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:16.356594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.356503 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" Nov 03 08:25:16.356594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.356524 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6" event={"ID":"61ecfdea-2575-4a74-aeed-2a9068ae0d0f","Type":"ContainerDied","Data":"b6f40241061fa87d09c137cf234aac3e39fe0eb22009e3518a55c2a27e4a9165"} Nov 03 08:25:16.356594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.356585 5951 scope.go:122] "RemoveContainer" containerID="8ba2726ae3299ec24374c8489192106b95b305a9e21f39d2201a368fa17aef01" Nov 03 08:25:16.358472 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.358447 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" Nov 03 08:25:16.358582 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.358504 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb" event={"ID":"aec249df-0e55-474e-ad96-e8a0769fd2bd","Type":"ContainerDied","Data":"f70a8efcf0078985ca45d21bb27ce6d8b353e278dd19f3f361043497398b87de"} Nov 03 08:25:16.368540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.368508 5951 scope.go:122] "RemoveContainer" containerID="89d7a7844e2224149fca5e7a8a6473ae21b50bceabf7f15751641a830112566d" Nov 03 08:25:16.379783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.379743 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6"] Nov 03 08:25:16.383747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.383724 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-bd085d-79b8b865cf-8jtw6"] Nov 03 08:25:16.389925 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.389901 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb"] Nov 03 08:25:16.393610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.393588 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-bd085d-c8bb6c5d9-sb8kb"] Nov 03 08:25:16.739467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.739376 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" path="/var/lib/kubelet/pods/61ecfdea-2575-4a74-aeed-2a9068ae0d0f/volumes" Nov 03 08:25:16.740100 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:16.740074 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" path="/var/lib/kubelet/pods/aec249df-0e55-474e-ad96-e8a0769fd2bd/volumes" Nov 03 08:25:18.153945 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:18.153907 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42"] Nov 03 08:25:18.154455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:18.154146 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" containerID="cri-o://7d3c1979df6083c862845b52bf6454ca90f33a025f8ef8ba6e2dc6a5352773ae" gracePeriod=30 Nov 03 08:25:23.441798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.441706 5951 generic.go:356] "Generic (PLEG): container finished" podID="fd997040-85b1-4ebf-be01-04de25803871" containerID="7d3c1979df6083c862845b52bf6454ca90f33a025f8ef8ba6e2dc6a5352773ae" exitCode=0 Nov 03 08:25:23.442289 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.441887 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" event={"ID":"fd997040-85b1-4ebf-be01-04de25803871","Type":"ContainerDied","Data":"7d3c1979df6083c862845b52bf6454ca90f33a025f8ef8ba6e2dc6a5352773ae"} Nov 03 08:25:23.485966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.485938 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:25:23.586411 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.586364 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.586628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.586443 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.586628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.586615 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.586809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.586706 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:23.586910 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.586855 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:23.587094 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587062 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:25:23.587215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587198 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.587708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587354 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.587708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587405 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.587708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587440 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.587708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587535 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.587708 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587567 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data\") pod \"fd997040-85b1-4ebf-be01-04de25803871\" (UID: \"fd997040-85b1-4ebf-be01-04de25803871\") " Nov 03 08:25:23.588063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.587779 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:23.588063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.588020 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:23.588176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.588154 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd997040-85b1-4ebf-be01-04de25803871-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.588176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.588176 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.588350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.588191 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.588350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.588204 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.588350 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.588216 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.589952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.589924 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:25:23.590304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.590283 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "kube-api-access-98tmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:23.590417 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.590400 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:25:23.590590 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.590572 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token" pod "fd997040-85b1-4ebf-be01-04de25803871" (UID: "fd997040-85b1-4ebf-be01-04de25803871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:25:23.689184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.689142 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98tmz\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-kube-api-access-98tmz\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.689184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.689180 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd997040-85b1-4ebf-be01-04de25803871-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.689184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.689191 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd997040-85b1-4ebf-be01-04de25803871-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:23.689443 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:23.689201 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd997040-85b1-4ebf-be01-04de25803871-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:25:24.452262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:24.452230 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" Nov 03 08:25:24.452262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:24.452254 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42" event={"ID":"fd997040-85b1-4ebf-be01-04de25803871","Type":"ContainerDied","Data":"f978b80dcb0e1611079ee454647b283e0ca0d3407c483c20b461e49f4b1cac76"} Nov 03 08:25:24.452874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:24.452308 5951 scope.go:122] "RemoveContainer" containerID="7d3c1979df6083c862845b52bf6454ca90f33a025f8ef8ba6e2dc6a5352773ae" Nov 03 08:25:24.487080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:24.487044 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42"] Nov 03 08:25:24.491041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:24.491014 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-5dg42"] Nov 03 08:25:24.737447 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:25:24.737356 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd997040-85b1-4ebf-be01-04de25803871" path="/var/lib/kubelet/pods/fd997040-85b1-4ebf-be01-04de25803871/volumes" Nov 03 08:25:31.956382 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:25:31.956334 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:26:01.954247 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:26:01.954211 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:26:31.958379 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:26:31.958333 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:26:48.041470 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:26:48.041429 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z"] Nov 03 08:26:48.047340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:26:48.047303 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e7ab602-ct76z"] Nov 03 08:26:48.737528 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:26:48.737495 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96897ed3-96a4-46a1-87aa-99333dae25eb" path="/var/lib/kubelet/pods/96897ed3-96a4-46a1-87aa-99333dae25eb/volumes" Nov 03 08:27:01.463465 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.463423 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp"] Nov 03 08:27:01.464386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.464349 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" podUID="68cc6679-e4d0-4740-bbac-233ed50166fa" containerName="maas-api" containerID="cri-o://c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30" gracePeriod=30 Nov 03 08:27:01.838073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.838046 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:27:01.943996 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.943953 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp\" (UniqueName: \"kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp\") pod \"68cc6679-e4d0-4740-bbac-233ed50166fa\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " Nov 03 08:27:01.944178 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.944162 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca\") pod \"68cc6679-e4d0-4740-bbac-233ed50166fa\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " Nov 03 08:27:01.944234 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.944196 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs\") pod \"68cc6679-e4d0-4740-bbac-233ed50166fa\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " Nov 03 08:27:01.944234 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.944229 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\") pod \"68cc6679-e4d0-4740-bbac-233ed50166fa\" (UID: \"68cc6679-e4d0-4740-bbac-233ed50166fa\") " Nov 03 08:27:01.944975 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.944905 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca" pod "68cc6679-e4d0-4740-bbac-233ed50166fa" (UID: "68cc6679-e4d0-4740-bbac-233ed50166fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:01.951114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.951079 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs" pod "68cc6679-e4d0-4740-bbac-233ed50166fa" (UID: "68cc6679-e4d0-4740-bbac-233ed50166fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:01.953128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.953084 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls" pod "68cc6679-e4d0-4740-bbac-233ed50166fa" (UID: "68cc6679-e4d0-4740-bbac-233ed50166fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:01.953889 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:01.953864 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp" pod "68cc6679-e4d0-4740-bbac-233ed50166fa" (UID: "68cc6679-e4d0-4740-bbac-233ed50166fa"). InnerVolumeSpecName "kube-api-access-xnjdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:01.962850 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:01.962803 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:27:02.045848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.045805 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjdp\" (UniqueName: \"kubernetes.io/projected/68cc6679-e4d0-4740-bbac-233ed50166fa-kube-api-access-xnjdp\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:02.045848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.045844 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cc6679-e4d0-4740-bbac-233ed50166fa-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:02.046081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.045860 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:02.046081 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.045875 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68cc6679-e4d0-4740-bbac-233ed50166fa-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:02.283420 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.283386 5951 generic.go:356] "Generic (PLEG): container finished" podID="68cc6679-e4d0-4740-bbac-233ed50166fa" containerID="c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30" exitCode=0 Nov 03 08:27:02.283602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.283436 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" Nov 03 08:27:02.283602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.283469 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" event={"ID":"68cc6679-e4d0-4740-bbac-233ed50166fa","Type":"ContainerDied","Data":"c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30"} Nov 03 08:27:02.283602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.283525 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp" event={"ID":"68cc6679-e4d0-4740-bbac-233ed50166fa","Type":"ContainerDied","Data":"7d6395185e52907957153763534c69007d0b37c5e6676a809685842810794b6e"} Nov 03 08:27:02.283602 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.283548 5951 scope.go:122] "RemoveContainer" containerID="c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30" Nov 03 08:27:02.301476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.301449 5951 scope.go:122] "RemoveContainer" containerID="c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30" Nov 03 08:27:02.302106 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:02.302074 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30\": container with ID starting with c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30 not found: ID does not exist" containerID="c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30" Nov 03 08:27:02.302227 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.302121 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30"} err="failed to get container status \"c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30\": rpc error: code = NotFound desc = could not find container \"c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30\": container with ID starting with c12eebaeae25aaa1f68b73fa8977452a17c33a59a72c75a51f049a64563d9f30 not found: ID does not exist" Nov 03 08:27:02.313247 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.313205 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp"] Nov 03 08:27:02.316636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.316607 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a61a48-7c6db8bbcb-scjdp"] Nov 03 08:27:02.737957 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:02.737869 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cc6679-e4d0-4740-bbac-233ed50166fa" path="/var/lib/kubelet/pods/68cc6679-e4d0-4740-bbac-233ed50166fa/volumes" Nov 03 08:27:20.168676 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:20.168625 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx"] Nov 03 08:27:24.827716 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:24.827678 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps"] Nov 03 08:27:24.828195 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:24.827962 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" podUID="1a01dbb7-a4c7-4393-9da1-72e7930db782" containerName="maas-api" containerID="cri-o://713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1" gracePeriod=30 Nov 03 08:27:25.169190 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.168961 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerName="payload-pre-processing" containerID="cri-o://a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1" gracePeriod=25 Nov 03 08:27:25.208278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.208251 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:27:25.303780 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.303739 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls\") pod \"1a01dbb7-a4c7-4393-9da1-72e7930db782\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " Nov 03 08:27:25.303985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.303837 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs\") pod \"1a01dbb7-a4c7-4393-9da1-72e7930db782\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " Nov 03 08:27:25.303985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.303920 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch\" (UniqueName: \"kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch\") pod \"1a01dbb7-a4c7-4393-9da1-72e7930db782\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " Nov 03 08:27:25.304110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.304040 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca\") pod \"1a01dbb7-a4c7-4393-9da1-72e7930db782\" (UID: \"1a01dbb7-a4c7-4393-9da1-72e7930db782\") " Nov 03 08:27:25.304423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.304399 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca" pod "1a01dbb7-a4c7-4393-9da1-72e7930db782" (UID: "1a01dbb7-a4c7-4393-9da1-72e7930db782"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:25.305063 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.305039 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01dbb7-a4c7-4393-9da1-72e7930db782-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:25.310096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.306636 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch" pod "1a01dbb7-a4c7-4393-9da1-72e7930db782" (UID: "1a01dbb7-a4c7-4393-9da1-72e7930db782"). InnerVolumeSpecName "kube-api-access-9lqch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:25.310096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.306743 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs" pod "1a01dbb7-a4c7-4393-9da1-72e7930db782" (UID: "1a01dbb7-a4c7-4393-9da1-72e7930db782"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:25.310096 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.306755 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls" pod "1a01dbb7-a4c7-4393-9da1-72e7930db782" (UID: "1a01dbb7-a4c7-4393-9da1-72e7930db782"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:25.406013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.405957 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lqch\" (UniqueName: \"kubernetes.io/projected/1a01dbb7-a4c7-4393-9da1-72e7930db782-kube-api-access-9lqch\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:25.406013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.405997 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:25.406013 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.406012 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01dbb7-a4c7-4393-9da1-72e7930db782-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:25.475061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.474953 5951 generic.go:356] "Generic (PLEG): container finished" podID="1a01dbb7-a4c7-4393-9da1-72e7930db782" containerID="713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1" exitCode=0 Nov 03 08:27:25.475061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.475012 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" Nov 03 08:27:25.475061 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.475010 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" event={"ID":"1a01dbb7-a4c7-4393-9da1-72e7930db782","Type":"ContainerDied","Data":"713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1"} Nov 03 08:27:25.475330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.475096 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps" event={"ID":"1a01dbb7-a4c7-4393-9da1-72e7930db782","Type":"ContainerDied","Data":"122cc9175deac2466ac2b349d5f6b1be88e7d7bc86915cd8dac291e03d517436"} Nov 03 08:27:25.475330 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.475145 5951 scope.go:122] "RemoveContainer" containerID="713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1" Nov 03 08:27:25.489579 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.489558 5951 scope.go:122] "RemoveContainer" containerID="713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1" Nov 03 08:27:25.489980 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:25.489958 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1\": container with ID starting with 713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1 not found: ID does not exist" containerID="713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1" Nov 03 08:27:25.490091 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.489986 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1"} err="failed to get container status \"713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1\": rpc error: code = NotFound desc = could not find container \"713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1\": container with ID starting with 713cfc8d7bd6741d100652231bcde1b2957137d3c1300ae6ee64f438c39dc5d1 not found: ID does not exist" Nov 03 08:27:25.500462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.500427 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps"] Nov 03 08:27:25.503968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:25.503945 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6a8bf-845bb55667-nc4ps"] Nov 03 08:27:26.062262 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:26.062230 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw"] Nov 03 08:27:26.063125 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:26.062451 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" containerID="cri-o://f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e" gracePeriod=30 Nov 03 08:27:26.737437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:26.737402 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a01dbb7-a4c7-4393-9da1-72e7930db782" path="/var/lib/kubelet/pods/1a01dbb7-a4c7-4393-9da1-72e7930db782/volumes" Nov 03 08:27:27.978568 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:27.978527 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:27:30.210271 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:30.210224 5951 patch_prober.go:29] interesting pod/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:27:30.210715 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:30.210298 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:27:31.349994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.349928 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:27:31.476871 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.476839 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.476907 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.476963 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477065 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477028 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477110 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477133 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:31.477241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477177 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477229 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477268 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477300 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket\") pod \"5a351ffb-58a7-464c-a905-bcbfe06847ba\" (UID: \"5a351ffb-58a7-464c-a905-bcbfe06847ba\") " Nov 03 08:27:31.477448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477397 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:31.477613 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477437 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:31.477697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477632 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:31.477764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.477731 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:31.478140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.478109 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a351ffb-58a7-464c-a905-bcbfe06847ba-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.478213 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.478138 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.478213 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.478156 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.478213 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.478170 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.478213 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.478184 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.479466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.479445 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:31.479466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.479454 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:27:31.479873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.479855 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:31.479958 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.479894 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt" pod "5a351ffb-58a7-464c-a905-bcbfe06847ba" (UID: "5a351ffb-58a7-464c-a905-bcbfe06847ba"). InnerVolumeSpecName "kube-api-access-r24xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:31.522739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.522702 5951 generic.go:356] "Generic (PLEG): container finished" podID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerID="f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e" exitCode=0 Nov 03 08:27:31.522917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.522767 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" Nov 03 08:27:31.522917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.522771 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" event={"ID":"5a351ffb-58a7-464c-a905-bcbfe06847ba","Type":"ContainerDied","Data":"f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e"} Nov 03 08:27:31.522917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.522842 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw" event={"ID":"5a351ffb-58a7-464c-a905-bcbfe06847ba","Type":"ContainerDied","Data":"8330da983515ffab758ee90d4fa155d85f5b14583a795a86d414be8ad0348278"} Nov 03 08:27:31.522917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.522871 5951 scope.go:122] "RemoveContainer" containerID="f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e" Nov 03 08:27:31.550673 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.550610 5951 scope.go:122] "RemoveContainer" containerID="1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610" Nov 03 08:27:31.555628 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.555580 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw"] Nov 03 08:27:31.562286 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.562253 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a66d5c-openshift-default-9b99ccc77-5qzvw"] Nov 03 08:27:31.563201 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.563176 5951 scope.go:122] "RemoveContainer" containerID="f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e" Nov 03 08:27:31.563478 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:31.563457 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e\": container with ID starting with f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e not found: ID does not exist" containerID="f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e" Nov 03 08:27:31.563569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.563491 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e"} err="failed to get container status \"f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e\": rpc error: code = NotFound desc = could not find container \"f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e\": container with ID starting with f5c2ca9413a3dda6b872696167f30ab09abdfdcbb17756125b8adb6f430c803e not found: ID does not exist" Nov 03 08:27:31.563569 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.563517 5951 scope.go:122] "RemoveContainer" containerID="1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610" Nov 03 08:27:31.563833 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:31.563811 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610\": container with ID starting with 1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610 not found: ID does not exist" containerID="1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610" Nov 03 08:27:31.563893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.563844 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610"} err="failed to get container status \"1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610\": rpc error: code = NotFound desc = could not find container \"1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610\": container with ID starting with 1b12f93f97b23a008c530573cd1b5f7004d6f87b57d166da29fb4709a506b610 not found: ID does not exist" Nov 03 08:27:31.579611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.579588 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24xt\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-kube-api-access-r24xt\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.579713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.579613 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.579713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.579625 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.579713 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:31.579633 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a351ffb-58a7-464c-a905-bcbfe06847ba-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:31.959856 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:31.959815 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:27:32.607392 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:32.607351 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc"] Nov 03 08:27:32.607931 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:32.607662 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" containerID="cri-o://3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84" gracePeriod=30 Nov 03 08:27:32.745863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:32.745820 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" path="/var/lib/kubelet/pods/5a351ffb-58a7-464c-a905-bcbfe06847ba/volumes" Nov 03 08:27:33.633473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:33.633135 5951 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:27:33.633473 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:33.633222 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:27:37.934072 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:37.934045 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:27:37.978423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:37.978380 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:27:38.009266 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009216 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009308 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009348 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009402 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009434 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009488 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009570 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009617 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.009751 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.009717 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data\") pod \"3c6aa698-50f9-4827-bfb2-3af5344613cd\" (UID: \"3c6aa698-50f9-4827-bfb2-3af5344613cd\") " Nov 03 08:27:38.010560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.010505 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:38.010811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.010790 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:38.011462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.011131 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:38.011681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.011629 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:38.011861 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.011838 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:38.017288 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.017262 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "kube-api-access-ggx5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:38.017404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.017298 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:38.017752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.017727 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:27:38.025310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.025261 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy" pod "3c6aa698-50f9-4827-bfb2-3af5344613cd" (UID: "3c6aa698-50f9-4827-bfb2-3af5344613cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:38.043066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.043028 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv"] Nov 03 08:27:38.058396 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.058362 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq"] Nov 03 08:27:38.112120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112072 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112111 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112120 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112126 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112140 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6aa698-50f9-4827-bfb2-3af5344613cd-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112153 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112168 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112181 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggx5f\" (UniqueName: \"kubernetes.io/projected/3c6aa698-50f9-4827-bfb2-3af5344613cd-kube-api-access-ggx5f\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112196 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.112439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.112209 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6aa698-50f9-4827-bfb2-3af5344613cd-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:38.576484 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.576449 5951 generic.go:356] "Generic (PLEG): container finished" podID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerID="3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84" exitCode=0 Nov 03 08:27:38.576741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.576523 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" Nov 03 08:27:38.576741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.576534 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" event={"ID":"3c6aa698-50f9-4827-bfb2-3af5344613cd","Type":"ContainerDied","Data":"3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84"} Nov 03 08:27:38.576741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.576632 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc" event={"ID":"3c6aa698-50f9-4827-bfb2-3af5344613cd","Type":"ContainerDied","Data":"ae8f2c3c6ac46cc5c1b328b9ccc44691ba306dcd0077119241d20c4adb7a4a66"} Nov 03 08:27:38.576741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.576682 5951 scope.go:122] "RemoveContainer" containerID="3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84" Nov 03 08:27:38.587305 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.587283 5951 scope.go:122] "RemoveContainer" containerID="3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad" Nov 03 08:27:38.597976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.597949 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc"] Nov 03 08:27:38.598256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.598231 5951 scope.go:122] "RemoveContainer" containerID="3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84" Nov 03 08:27:38.598544 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:38.598526 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84\": container with ID starting with 3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84 not found: ID does not exist" containerID="3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84" Nov 03 08:27:38.598603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.598552 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84"} err="failed to get container status \"3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84\": rpc error: code = NotFound desc = could not find container \"3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84\": container with ID starting with 3bec78906261b037a64bcaae816b2978322f384cf4221aa85f95aa4e70578a84 not found: ID does not exist" Nov 03 08:27:38.598603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.598570 5951 scope.go:122] "RemoveContainer" containerID="3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad" Nov 03 08:27:38.598874 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:38.598851 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad\": container with ID starting with 3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad not found: ID does not exist" containerID="3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad" Nov 03 08:27:38.598941 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.598885 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad"} err="failed to get container status \"3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad\": rpc error: code = NotFound desc = could not find container \"3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad\": container with ID starting with 3a9ed6d449d8384cd287c56b7760166feedd53caf68fb9800b48812bd65344ad not found: ID does not exist" Nov 03 08:27:38.604101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.604069 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d6a8bf-openshift-default-5b54849f9c-zd4qc"] Nov 03 08:27:38.740523 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:38.740482 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" path="/var/lib/kubelet/pods/3c6aa698-50f9-4827-bfb2-3af5344613cd/volumes" Nov 03 08:27:43.378537 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:43.378495 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8"] Nov 03 08:27:45.447936 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.447871 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:27:45.513669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.507110 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume\") pod \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " Nov 03 08:27:45.514040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.513997 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw\" (UniqueName: \"kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw\") pod \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\" (UID: \"1030fc0d-5e30-47a0-bd3e-9640d42e781c\") " Nov 03 08:27:45.529395 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.519453 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume" pod "1030fc0d-5e30-47a0-bd3e-9640d42e781c" (UID: "1030fc0d-5e30-47a0-bd3e-9640d42e781c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:45.533272 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.533221 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw" pod "1030fc0d-5e30-47a0-bd3e-9640d42e781c" (UID: "1030fc0d-5e30-47a0-bd3e-9640d42e781c"). InnerVolumeSpecName "kube-api-access-m94sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:45.631018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.630970 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1030fc0d-5e30-47a0-bd3e-9640d42e781c-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:45.631018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.631008 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94sw\" (UniqueName: \"kubernetes.io/projected/1030fc0d-5e30-47a0-bd3e-9640d42e781c-kube-api-access-m94sw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:45.641162 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.641105 5951 generic.go:356] "Generic (PLEG): container finished" podID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerID="a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1" exitCode=0 Nov 03 08:27:45.641349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.641214 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" event={"ID":"1030fc0d-5e30-47a0-bd3e-9640d42e781c","Type":"ContainerDied","Data":"a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1"} Nov 03 08:27:45.641349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.641268 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" event={"ID":"1030fc0d-5e30-47a0-bd3e-9640d42e781c","Type":"ContainerDied","Data":"005e00f60920395b22c5c4ecc37bbbb4a8e3a7c3c799ab3b27abaddea7dae6b3"} Nov 03 08:27:45.641349 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.641298 5951 scope.go:122] "RemoveContainer" containerID="a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1" Nov 03 08:27:45.641532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.641445 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx" Nov 03 08:27:45.668896 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.668867 5951 scope.go:122] "RemoveContainer" containerID="a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1" Nov 03 08:27:45.669257 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:45.669227 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1\": container with ID starting with a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1 not found: ID does not exist" containerID="a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1" Nov 03 08:27:45.669348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.669270 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1"} err="failed to get container status \"a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1\": rpc error: code = NotFound desc = could not find container \"a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1\": container with ID starting with a778df83e55b5c89dba67c470f4f2597bf3a201451bf5e201ac94892d43cdeb1 not found: ID does not exist" Nov 03 08:27:45.680424 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.680387 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx"] Nov 03 08:27:45.685681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:45.685620 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d6a8bf-774b49dc75-4xfgx"] Nov 03 08:27:46.735897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:46.735861 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" path="/var/lib/kubelet/pods/1030fc0d-5e30-47a0-bd3e-9640d42e781c/volumes" Nov 03 08:27:48.378890 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:48.378815 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerName="payload-pre-processing" containerID="cri-o://b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70" gracePeriod=25 Nov 03 08:27:50.624596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:50.624538 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w"] Nov 03 08:27:50.625129 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:50.624878 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" containerID="cri-o://cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5" gracePeriod=30 Nov 03 08:27:53.067285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.067221 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" containerID="cri-o://c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884" gracePeriod=45 Nov 03 08:27:53.083165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.083124 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" containerID="cri-o://1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94" gracePeriod=45 Nov 03 08:27:53.407656 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.405041 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:27:53.407656 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.407296 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:27:53.544214 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544183 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache\") pod \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " Nov 03 08:27:53.544404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544234 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7\" (UniqueName: \"kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7\") pod \"1ca0f984-9823-43bd-a10e-6fd21931b505\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " Nov 03 08:27:53.544404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544263 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm\") pod \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " Nov 03 08:27:53.544404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544343 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home\") pod \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " Nov 03 08:27:53.544404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544382 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs\") pod \"1ca0f984-9823-43bd-a10e-6fd21931b505\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " Nov 03 08:27:53.544404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544399 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home\") pod \"1ca0f984-9823-43bd-a10e-6fd21931b505\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544425 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir\") pod \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544442 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc\" (UniqueName: \"kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc\") pod \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544472 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir\") pod \"1ca0f984-9823-43bd-a10e-6fd21931b505\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544493 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache" pod "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" (UID: "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544515 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache\") pod \"1ca0f984-9823-43bd-a10e-6fd21931b505\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544600 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs\") pod \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\" (UID: \"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544628 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm\") pod \"1ca0f984-9823-43bd-a10e-6fd21931b505\" (UID: \"1ca0f984-9823-43bd-a10e-6fd21931b505\") " Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.544788 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home" pod "1ca0f984-9823-43bd-a10e-6fd21931b505" (UID: "1ca0f984-9823-43bd-a10e-6fd21931b505"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545001 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir" pod "1ca0f984-9823-43bd-a10e-6fd21931b505" (UID: "1ca0f984-9823-43bd-a10e-6fd21931b505"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545015 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache" pod "1ca0f984-9823-43bd-a10e-6fd21931b505" (UID: "1ca0f984-9823-43bd-a10e-6fd21931b505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545147 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home" pod "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" (UID: "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545223 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545236 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir" pod "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" (UID: "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545243 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545272 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.545514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.545285 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.547555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.547532 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7" pod "1ca0f984-9823-43bd-a10e-6fd21931b505" (UID: "1ca0f984-9823-43bd-a10e-6fd21931b505"). InnerVolumeSpecName "kube-api-access-vwvp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:53.547720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.547698 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm" pod "1ca0f984-9823-43bd-a10e-6fd21931b505" (UID: "1ca0f984-9823-43bd-a10e-6fd21931b505"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.547720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.547704 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs" pod "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" (UID: "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:53.547845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.547724 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs" pod "1ca0f984-9823-43bd-a10e-6fd21931b505" (UID: "1ca0f984-9823-43bd-a10e-6fd21931b505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:53.547845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.547771 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc" pod "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" (UID: "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0"). InnerVolumeSpecName "kube-api-access-tdxcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:53.549099 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.549080 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm" pod "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" (UID: "9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646775 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646806 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca0f984-9823-43bd-a10e-6fd21931b505-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646817 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvp7\" (UniqueName: \"kubernetes.io/projected/1ca0f984-9823-43bd-a10e-6fd21931b505-kube-api-access-vwvp7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646827 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646836 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646844 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca0f984-9823-43bd-a10e-6fd21931b505-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646851 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.646855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.646859 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxcc\" (UniqueName: \"kubernetes.io/projected/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0-kube-api-access-tdxcc\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:53.696810 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.696775 5951 generic.go:356] "Generic (PLEG): container finished" podID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerID="1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94" exitCode=0 Nov 03 08:27:53.696993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.696837 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" Nov 03 08:27:53.696993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.696866 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" event={"ID":"1ca0f984-9823-43bd-a10e-6fd21931b505","Type":"ContainerDied","Data":"1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94"} Nov 03 08:27:53.696993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.696919 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" event={"ID":"1ca0f984-9823-43bd-a10e-6fd21931b505","Type":"ContainerDied","Data":"bb40a3d6d0fd1aeae8c55db122b675315bcf48993e8e8a46bcc2f4fc38a78d07"} Nov 03 08:27:53.696993 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.696941 5951 scope.go:122] "RemoveContainer" containerID="1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94" Nov 03 08:27:53.698383 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.698358 5951 generic.go:356] "Generic (PLEG): container finished" podID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerID="c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884" exitCode=0 Nov 03 08:27:53.698512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.698394 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" Nov 03 08:27:53.698512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.698446 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" event={"ID":"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0","Type":"ContainerDied","Data":"c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884"} Nov 03 08:27:53.698512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.698487 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv" event={"ID":"9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0","Type":"ContainerDied","Data":"6453163236d056b05319b07f38d414af793beef5ea20140f8cebcbb8350ba028"} Nov 03 08:27:53.707679 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.707660 5951 scope.go:122] "RemoveContainer" containerID="1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94" Nov 03 08:27:53.707939 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:53.707920 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94\": container with ID starting with 1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94 not found: ID does not exist" containerID="1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94" Nov 03 08:27:53.708040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.707949 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94"} err="failed to get container status \"1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94\": rpc error: code = NotFound desc = could not find container \"1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94\": container with ID starting with 1704637df306b1ac446c61057443003a15fccc502981367b3e751fc894e97d94 not found: ID does not exist" Nov 03 08:27:53.708040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.707968 5951 scope.go:122] "RemoveContainer" containerID="c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884" Nov 03 08:27:53.717028 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.717007 5951 scope.go:122] "RemoveContainer" containerID="c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884" Nov 03 08:27:53.717305 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:53.717285 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884\": container with ID starting with c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884 not found: ID does not exist" containerID="c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884" Nov 03 08:27:53.717375 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.717314 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884"} err="failed to get container status \"c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884\": rpc error: code = NotFound desc = could not find container \"c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884\": container with ID starting with c585cf9418ad99ebd7ea7ae58fdb74aef825cd34348e937eafe6b9a241f83884 not found: ID does not exist" Nov 03 08:27:53.748994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.748957 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq"] Nov 03 08:27:53.759207 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.759179 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq"] Nov 03 08:27:53.774197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.774159 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv"] Nov 03 08:27:53.777712 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:53.777687 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-2-w1-19bb0a-kserve-8487c966b6-nltrv"] Nov 03 08:27:54.316636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:54.316594 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/premium-w1-19bb0a-kserve-685f6c7566-sqlqq" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:27:54.736859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:54.736822 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" path="/var/lib/kubelet/pods/1ca0f984-9823-43bd-a10e-6fd21931b505/volumes" Nov 03 08:27:54.737335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:54.737319 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" path="/var/lib/kubelet/pods/9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0/volumes" Nov 03 08:27:55.922215 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.922187 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:27:55.975851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.975818 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.975895 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.975997 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976049 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976023 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976089 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976124 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976169 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976229 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976300 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data\") pod \"901fa1d8-7cb9-4d66-8a68-36970e80813a\" (UID: \"901fa1d8-7cb9-4d66-8a68-36970e80813a\") " Nov 03 08:27:55.976439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976340 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:55.976439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976412 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:55.976603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976522 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:55.976748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976713 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:55.976748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976736 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:55.976748 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976751 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:55.977040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976780 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:55.977040 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.976831 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:55.978851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.978826 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "kube-api-access-qph5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:55.978963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.978944 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:27:55.979190 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.979170 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:55.979190 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:55.979173 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy" pod "901fa1d8-7cb9-4d66-8a68-36970e80813a" (UID: "901fa1d8-7cb9-4d66-8a68-36970e80813a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:27:56.077863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.077812 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:56.077863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.077859 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:56.078070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.077874 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:56.078070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.077887 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/901fa1d8-7cb9-4d66-8a68-36970e80813a-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:56.078070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.077901 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/901fa1d8-7cb9-4d66-8a68-36970e80813a-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:56.078070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.077915 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qph5t\" (UniqueName: \"kubernetes.io/projected/901fa1d8-7cb9-4d66-8a68-36970e80813a-kube-api-access-qph5t\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:56.725041 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.725000 5951 generic.go:356] "Generic (PLEG): container finished" podID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerID="cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5" exitCode=0 Nov 03 08:27:56.725216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.725072 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" Nov 03 08:27:56.725216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.725097 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" event={"ID":"901fa1d8-7cb9-4d66-8a68-36970e80813a","Type":"ContainerDied","Data":"cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5"} Nov 03 08:27:56.725216 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.725200 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w" event={"ID":"901fa1d8-7cb9-4d66-8a68-36970e80813a","Type":"ContainerDied","Data":"ff2778ca89f62edd478a3daeb8caed341048fcf18ca90e50c97aee5c1ee375a3"} Nov 03 08:27:56.725338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.725232 5951 scope.go:122] "RemoveContainer" containerID="cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5" Nov 03 08:27:56.738034 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.738012 5951 scope.go:122] "RemoveContainer" containerID="cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5" Nov 03 08:27:56.738343 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:56.738321 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5\": container with ID starting with cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5 not found: ID does not exist" containerID="cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5" Nov 03 08:27:56.738442 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.738354 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5"} err="failed to get container status \"cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5\": rpc error: code = NotFound desc = could not find container \"cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5\": container with ID starting with cd54da46ca883e86352e996a268ee7c11166e664c6fdbee46f7e0246f8e401b5 not found: ID does not exist" Nov 03 08:27:56.750681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.750635 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w"] Nov 03 08:27:56.753966 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:56.753933 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-xjc4w"] Nov 03 08:27:57.779179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:57.779140 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:27:58.736366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:58.736330 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" path="/var/lib/kubelet/pods/901fa1d8-7cb9-4d66-8a68-36970e80813a/volumes" Nov 03 08:27:59.254538 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.254503 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5"] Nov 03 08:27:59.255046 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.254812 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" podUID="cdcdb196-e296-4f90-95c7-fb1136ac2885" containerName="maas-api" containerID="cri-o://0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be" gracePeriod=30 Nov 03 08:27:59.772632 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.772607 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:27:59.824455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.824357 5951 generic.go:356] "Generic (PLEG): container finished" podID="cdcdb196-e296-4f90-95c7-fb1136ac2885" containerID="0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be" exitCode=0 Nov 03 08:27:59.824615 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.824573 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" event={"ID":"cdcdb196-e296-4f90-95c7-fb1136ac2885","Type":"ContainerDied","Data":"0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be"} Nov 03 08:27:59.824737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.824627 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" event={"ID":"cdcdb196-e296-4f90-95c7-fb1136ac2885","Type":"ContainerDied","Data":"0b78893f60adf1c98ae48340da9fe21b92ee71e3c35581cfc1b6ee147ea42a10"} Nov 03 08:27:59.824737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.824715 5951 scope.go:122] "RemoveContainer" containerID="0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be" Nov 03 08:27:59.824858 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.824842 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5" Nov 03 08:27:59.838677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.833799 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls\") pod \"cdcdb196-e296-4f90-95c7-fb1136ac2885\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " Nov 03 08:27:59.838677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.833889 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca\") pod \"cdcdb196-e296-4f90-95c7-fb1136ac2885\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " Nov 03 08:27:59.838677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.834017 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z\" (UniqueName: \"kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z\") pod \"cdcdb196-e296-4f90-95c7-fb1136ac2885\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " Nov 03 08:27:59.838677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.834053 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs\") pod \"cdcdb196-e296-4f90-95c7-fb1136ac2885\" (UID: \"cdcdb196-e296-4f90-95c7-fb1136ac2885\") " Nov 03 08:27:59.838677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.836786 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca" pod "cdcdb196-e296-4f90-95c7-fb1136ac2885" (UID: "cdcdb196-e296-4f90-95c7-fb1136ac2885"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:27:59.839115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.838875 5951 scope.go:122] "RemoveContainer" containerID="0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be" Nov 03 08:27:59.839241 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:27:59.839215 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be\": container with ID starting with 0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be not found: ID does not exist" containerID="0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be" Nov 03 08:27:59.839298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.839253 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be"} err="failed to get container status \"0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be\": rpc error: code = NotFound desc = could not find container \"0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be\": container with ID starting with 0839dd0baceb386086ccd88ad1e1c4dcd8713b1da1caec09129f22c0122a45be not found: ID does not exist" Nov 03 08:27:59.841243 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.841221 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls" pod "cdcdb196-e296-4f90-95c7-fb1136ac2885" (UID: "cdcdb196-e296-4f90-95c7-fb1136ac2885"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:59.842234 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.842207 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z" pod "cdcdb196-e296-4f90-95c7-fb1136ac2885" (UID: "cdcdb196-e296-4f90-95c7-fb1136ac2885"). InnerVolumeSpecName "kube-api-access-h6d9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:27:59.842423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.842399 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs" pod "cdcdb196-e296-4f90-95c7-fb1136ac2885" (UID: "cdcdb196-e296-4f90-95c7-fb1136ac2885"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:27:59.936836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.936797 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6d9z\" (UniqueName: \"kubernetes.io/projected/cdcdb196-e296-4f90-95c7-fb1136ac2885-kube-api-access-h6d9z\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:59.936836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.936826 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:59.936836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.936836 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcdb196-e296-4f90-95c7-fb1136ac2885-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:27:59.937077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:27:59.936844 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcdb196-e296-4f90-95c7-fb1136ac2885-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:00.148471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:00.148436 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5"] Nov 03 08:28:00.152807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:00.152781 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0fc79b-9f9546fbc-llpq5"] Nov 03 08:28:00.736514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:00.736482 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcdb196-e296-4f90-95c7-fb1136ac2885" path="/var/lib/kubelet/pods/cdcdb196-e296-4f90-95c7-fb1136ac2885/volumes" Nov 03 08:28:01.956095 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:01.956056 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:28:05.077905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.077867 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc"] Nov 03 08:28:05.078455 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.078101 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" podUID="a64b095c-8435-44d7-8ca8-dab13c1f7aff" containerName="maas-api" containerID="cri-o://df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e" gracePeriod=30 Nov 03 08:28:05.400916 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.400891 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:28:05.468410 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.468372 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca\") pod \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " Nov 03 08:28:05.468584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.468496 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs\") pod \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " Nov 03 08:28:05.468584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.468534 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx\" (UniqueName: \"kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx\") pod \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " Nov 03 08:28:05.468584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.468575 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls\") pod \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\" (UID: \"a64b095c-8435-44d7-8ca8-dab13c1f7aff\") " Nov 03 08:28:05.468839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.468816 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca" pod "a64b095c-8435-44d7-8ca8-dab13c1f7aff" (UID: "a64b095c-8435-44d7-8ca8-dab13c1f7aff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:28:05.468951 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.468934 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64b095c-8435-44d7-8ca8-dab13c1f7aff-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:05.470948 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.470919 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs" pod "a64b095c-8435-44d7-8ca8-dab13c1f7aff" (UID: "a64b095c-8435-44d7-8ca8-dab13c1f7aff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:28:05.471057 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.471004 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx" pod "a64b095c-8435-44d7-8ca8-dab13c1f7aff" (UID: "a64b095c-8435-44d7-8ca8-dab13c1f7aff"). InnerVolumeSpecName "kube-api-access-2rttx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:05.471114 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.471068 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls" pod "a64b095c-8435-44d7-8ca8-dab13c1f7aff" (UID: "a64b095c-8435-44d7-8ca8-dab13c1f7aff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:28:05.571946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.571908 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:05.571946 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.571947 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rttx\" (UniqueName: \"kubernetes.io/projected/a64b095c-8435-44d7-8ca8-dab13c1f7aff-kube-api-access-2rttx\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:05.572198 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.571962 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a64b095c-8435-44d7-8ca8-dab13c1f7aff-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:05.889558 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.889525 5951 generic.go:356] "Generic (PLEG): container finished" podID="a64b095c-8435-44d7-8ca8-dab13c1f7aff" containerID="df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e" exitCode=0 Nov 03 08:28:05.889816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.889666 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" event={"ID":"a64b095c-8435-44d7-8ca8-dab13c1f7aff","Type":"ContainerDied","Data":"df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e"} Nov 03 08:28:05.889816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.889719 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" event={"ID":"a64b095c-8435-44d7-8ca8-dab13c1f7aff","Type":"ContainerDied","Data":"f17adda0a27d81db3af8442dc50efa722cb92206c93e0db43be3df9bab901173"} Nov 03 08:28:05.889816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.889747 5951 scope.go:122] "RemoveContainer" containerID="df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e" Nov 03 08:28:05.890000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.889892 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc" Nov 03 08:28:05.908386 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.908362 5951 scope.go:122] "RemoveContainer" containerID="df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e" Nov 03 08:28:05.908843 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:05.908781 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e\": container with ID starting with df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e not found: ID does not exist" containerID="df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e" Nov 03 08:28:05.908947 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.908853 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e"} err="failed to get container status \"df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e\": rpc error: code = NotFound desc = could not find container \"df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e\": container with ID starting with df061ce6e26a950e7d3046fb19a51a4876caab3d9ae17238a6a772b46f98ba8e not found: ID does not exist" Nov 03 08:28:05.912337 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.912309 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc"] Nov 03 08:28:05.915426 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:05.915402 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-178e68-65465d9c67-2p2tc"] Nov 03 08:28:06.740123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:06.740086 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64b095c-8435-44d7-8ca8-dab13c1f7aff" path="/var/lib/kubelet/pods/a64b095c-8435-44d7-8ca8-dab13c1f7aff/volumes" Nov 03 08:28:06.852345 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:06.852306 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn"] Nov 03 08:28:06.852675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:06.852576 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" containerID="cri-o://eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf" gracePeriod=30 Nov 03 08:28:07.735766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:07.735729 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:28:08.659327 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.659298 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:28:08.717250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.717194 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume\") pod \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " Nov 03 08:28:08.717451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.717343 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5\" (UniqueName: \"kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5\") pod \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\" (UID: \"c80ff06f-6f59-4664-9b6c-34a6272bf8eb\") " Nov 03 08:28:08.717740 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.717703 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume" pod "c80ff06f-6f59-4664-9b6c-34a6272bf8eb" (UID: "c80ff06f-6f59-4664-9b6c-34a6272bf8eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:28:08.722248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.722210 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5" pod "c80ff06f-6f59-4664-9b6c-34a6272bf8eb" (UID: "c80ff06f-6f59-4664-9b6c-34a6272bf8eb"). InnerVolumeSpecName "kube-api-access-b8wk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:08.818766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.818723 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:08.818766 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.818760 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wk5\" (UniqueName: \"kubernetes.io/projected/c80ff06f-6f59-4664-9b6c-34a6272bf8eb-kube-api-access-b8wk5\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:08.912589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.912489 5951 generic.go:356] "Generic (PLEG): container finished" podID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerID="b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70" exitCode=0 Nov 03 08:28:08.912589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.912544 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" Nov 03 08:28:08.912589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.912578 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" event={"ID":"c80ff06f-6f59-4664-9b6c-34a6272bf8eb","Type":"ContainerDied","Data":"b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70"} Nov 03 08:28:08.912933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.912636 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8" event={"ID":"c80ff06f-6f59-4664-9b6c-34a6272bf8eb","Type":"ContainerDied","Data":"dcc5021df4d798c923e231721ce9ae807536f9ff1604d6bc01c902ad37befe74"} Nov 03 08:28:08.912933 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.912681 5951 scope.go:122] "RemoveContainer" containerID="b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70" Nov 03 08:28:08.925219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.925196 5951 scope.go:122] "RemoveContainer" containerID="b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70" Nov 03 08:28:08.926552 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:08.925901 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70\": container with ID starting with b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70 not found: ID does not exist" containerID="b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70" Nov 03 08:28:08.926552 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.925943 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70"} err="failed to get container status \"b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70\": rpc error: code = NotFound desc = could not find container \"b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70\": container with ID starting with b50045a9beb63ac0458c7eb64730fd6af84290f40cbda9a56d3af7b555c70d70 not found: ID does not exist" Nov 03 08:28:08.928761 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.928731 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8"] Nov 03 08:28:08.932184 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:08.932157 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-19bb0a-6f85d57dc5-k68n8"] Nov 03 08:28:09.747236 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:09.747197 5951 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:28:09.747618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:09.747248 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:28:10.736761 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:10.736730 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" path="/var/lib/kubelet/pods/c80ff06f-6f59-4664-9b6c-34a6272bf8eb/volumes" Nov 03 08:28:12.182570 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.182478 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:28:12.276300 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276270 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276319 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276363 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276499 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276421 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276516 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276556 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276681 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276635 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276674 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:12.276859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276762 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.276859 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276805 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8\") pod \"dcaab243-c773-42cb-b917-d7ec04455bf1\" (UID: \"dcaab243-c773-42cb-b917-d7ec04455bf1\") " Nov 03 08:28:12.277012 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.276856 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:12.277128 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.277097 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:28:12.277203 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.277186 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.277263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.277210 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.277263 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.277224 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcaab243-c773-42cb-b917-d7ec04455bf1-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.277583 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.277551 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:12.277960 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.277629 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:12.279251 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.279213 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:12.279502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.279478 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "kube-api-access-hd4s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:12.279985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.279961 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:12.280068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.280002 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo" pod "dcaab243-c773-42cb-b917-d7ec04455bf1" (UID: "dcaab243-c773-42cb-b917-d7ec04455bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:28:12.378764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.378729 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.378764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.378761 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.378764 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.378771 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd4s8\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-kube-api-access-hd4s8\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.379009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.378781 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.379009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.378790 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.379009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.378798 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaab243-c773-42cb-b917-d7ec04455bf1-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:12.943745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.943705 5951 generic.go:356] "Generic (PLEG): container finished" podID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerID="eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf" exitCode=0 Nov 03 08:28:12.943918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.943754 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" Nov 03 08:28:12.943918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.943769 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" event={"ID":"dcaab243-c773-42cb-b917-d7ec04455bf1","Type":"ContainerDied","Data":"eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf"} Nov 03 08:28:12.943918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.943830 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn" event={"ID":"dcaab243-c773-42cb-b917-d7ec04455bf1","Type":"ContainerDied","Data":"f53e229e4adf031c1125741862116de07888ab83325a4b79baffd2e28e0988b2"} Nov 03 08:28:12.943918 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.943862 5951 scope.go:122] "RemoveContainer" containerID="eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf" Nov 03 08:28:12.954610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.954589 5951 scope.go:122] "RemoveContainer" containerID="eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf" Nov 03 08:28:12.954912 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:12.954887 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf\": container with ID starting with eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf not found: ID does not exist" containerID="eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf" Nov 03 08:28:12.954961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.954925 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf"} err="failed to get container status \"eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf\": rpc error: code = NotFound desc = could not find container \"eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf\": container with ID starting with eed3b543c442255db7a0de54f2345a67dd65b60031c9d39fa479574aecee53cf not found: ID does not exist" Nov 03 08:28:12.960459 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.960433 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn"] Nov 03 08:28:12.963601 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:12.963579 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-td9cn"] Nov 03 08:28:14.736976 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:14.736945 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" path="/var/lib/kubelet/pods/dcaab243-c773-42cb-b917-d7ec04455bf1/volumes" Nov 03 08:28:25.194921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:25.194876 5951 scope.go:122] "RemoveContainer" containerID="8843b5648ecd67b3f96b14ac21953ace83040bee4df544d83bafd2baa43cfdae" Nov 03 08:28:25.204886 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:25.204868 5951 scope.go:122] "RemoveContainer" containerID="d4f42ca9f59c04c08bc9dc02b5835d90badbe1ed56343e6dcada136b9411763b" Nov 03 08:28:26.999765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:26.999726 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb"] Nov 03 08:28:27.000589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:27.000555 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" containerID="cri-o://b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648" gracePeriod=30 Nov 03 08:28:31.953949 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:31.953902 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:28:32.276378 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.276346 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.310981 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311034 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311090 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311181 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311228 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311259 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311309 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311364 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311386 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data\") pod \"11d81aa6-a90e-4457-8dd9-41a95c107643\" (UID: \"11d81aa6-a90e-4457-8dd9-41a95c107643\") " Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.311886 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.312056 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.312554 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.313212 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:28:32.314192 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.313273 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:32.319973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.319933 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "kube-api-access-4x5mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:32.323194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.323163 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:28:32.323194 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.323180 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:32.325241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.325212 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token" pod "11d81aa6-a90e-4457-8dd9-41a95c107643" (UID: "11d81aa6-a90e-4457-8dd9-41a95c107643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:32.412299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412265 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412291 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412300 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412299 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412308 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412318 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412326 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d81aa6-a90e-4457-8dd9-41a95c107643-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412335 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x5mj\" (UniqueName: \"kubernetes.io/projected/11d81aa6-a90e-4457-8dd9-41a95c107643-kube-api-access-4x5mj\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412344 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d81aa6-a90e-4457-8dd9-41a95c107643-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:32.412574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:32.412353 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d81aa6-a90e-4457-8dd9-41a95c107643-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:33.083599 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.083560 5951 generic.go:356] "Generic (PLEG): container finished" podID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerID="b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648" exitCode=0 Nov 03 08:28:33.084142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.083653 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" Nov 03 08:28:33.084142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.083693 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" event={"ID":"11d81aa6-a90e-4457-8dd9-41a95c107643","Type":"ContainerDied","Data":"b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648"} Nov 03 08:28:33.084142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.083756 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" event={"ID":"11d81aa6-a90e-4457-8dd9-41a95c107643","Type":"ContainerDied","Data":"19005239c66489c14d9de2630a4c88db833361bee01d746093109abbfe0c2cec"} Nov 03 08:28:33.084142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.083787 5951 scope.go:122] "RemoveContainer" containerID="b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648" Nov 03 08:28:33.095749 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.095724 5951 scope.go:122] "RemoveContainer" containerID="c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f" Nov 03 08:28:33.100915 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.100892 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb"] Nov 03 08:28:33.104590 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.104563 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb"] Nov 03 08:28:33.107771 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.107755 5951 scope.go:122] "RemoveContainer" containerID="b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648" Nov 03 08:28:33.108047 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:33.108028 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648\": container with ID starting with b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648 not found: ID does not exist" containerID="b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648" Nov 03 08:28:33.108110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.108053 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648"} err="failed to get container status \"b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648\": rpc error: code = NotFound desc = could not find container \"b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648\": container with ID starting with b5b532bb7bab494663ca8dca127ae346056441bd008677d94f911a8a0cbf1648 not found: ID does not exist" Nov 03 08:28:33.108110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.108069 5951 scope.go:122] "RemoveContainer" containerID="c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f" Nov 03 08:28:33.108406 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:33.108376 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f\": container with ID starting with c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f not found: ID does not exist" containerID="c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f" Nov 03 08:28:33.108406 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.108400 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f"} err="failed to get container status \"c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f\": rpc error: code = NotFound desc = could not find container \"c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f\": container with ID starting with c8df3f28dfa3d36519170c174cdef9de8d28ea711164e8d2499017275d501a2f not found: ID does not exist" Nov 03 08:28:33.232352 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.232310 5951 patch_prober.go:29] interesting pod/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:28:33.232961 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:33.232923 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0d168e-openshift-default-7b88884569-98phb" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:28:34.736807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:34.736769 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" path="/var/lib/kubelet/pods/11d81aa6-a90e-4457-8dd9-41a95c107643/volumes" Nov 03 08:28:37.777625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:37.777586 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb"] Nov 03 08:28:38.304806 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:38.304768 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw"] Nov 03 08:28:42.778011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:42.777968 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerName="payload-pre-processing" containerID="cri-o://e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b" gracePeriod=25 Nov 03 08:28:43.602388 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:43.602345 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99"] Nov 03 08:28:44.673998 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:44.673954 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:28:48.602876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:48.602821 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerName="payload-processing" containerID="cri-o://8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665" gracePeriod=25 Nov 03 08:28:52.613874 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:52.613828 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:28:53.327239 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.327193 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" containerID="cri-o://2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6" gracePeriod=45 Nov 03 08:28:53.587618 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.587558 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:28:53.746772 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.746734 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir\") pod \"d8e89968-3a4b-4703-be77-07bfe931a99b\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " Nov 03 08:28:53.747174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.746800 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs\") pod \"d8e89968-3a4b-4703-be77-07bfe931a99b\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " Nov 03 08:28:53.747174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.746851 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm\") pod \"d8e89968-3a4b-4703-be77-07bfe931a99b\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " Nov 03 08:28:53.747174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.746906 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home\") pod \"d8e89968-3a4b-4703-be77-07bfe931a99b\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " Nov 03 08:28:53.747174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.746985 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5\" (UniqueName: \"kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5\") pod \"d8e89968-3a4b-4703-be77-07bfe931a99b\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " Nov 03 08:28:53.747174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.747049 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache\") pod \"d8e89968-3a4b-4703-be77-07bfe931a99b\" (UID: \"d8e89968-3a4b-4703-be77-07bfe931a99b\") " Nov 03 08:28:53.747523 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.747475 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home" pod "d8e89968-3a4b-4703-be77-07bfe931a99b" (UID: "d8e89968-3a4b-4703-be77-07bfe931a99b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:53.747625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.747549 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache" pod "d8e89968-3a4b-4703-be77-07bfe931a99b" (UID: "d8e89968-3a4b-4703-be77-07bfe931a99b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:53.747625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.747564 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir" pod "d8e89968-3a4b-4703-be77-07bfe931a99b" (UID: "d8e89968-3a4b-4703-be77-07bfe931a99b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:53.749850 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.749810 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs" pod "d8e89968-3a4b-4703-be77-07bfe931a99b" (UID: "d8e89968-3a4b-4703-be77-07bfe931a99b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:28:53.749985 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.749858 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm" pod "d8e89968-3a4b-4703-be77-07bfe931a99b" (UID: "d8e89968-3a4b-4703-be77-07bfe931a99b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:28:53.750301 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.750278 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5" pod "d8e89968-3a4b-4703-be77-07bfe931a99b" (UID: "d8e89968-3a4b-4703-be77-07bfe931a99b"). InnerVolumeSpecName "kube-api-access-8d2z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:53.848080 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.847987 5951 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-tmp-dir\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:53.848278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.848167 5951 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e89968-3a4b-4703-be77-07bfe931a99b-tls-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:53.848278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.848202 5951 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-dshm\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:53.848278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.848214 5951 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-home\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:53.848278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.848230 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2z5\" (UniqueName: \"kubernetes.io/projected/d8e89968-3a4b-4703-be77-07bfe931a99b-kube-api-access-8d2z5\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:53.848278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:53.848248 5951 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e89968-3a4b-4703-be77-07bfe931a99b-model-cache\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:54.223434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.223338 5951 generic.go:356] "Generic (PLEG): container finished" podID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerID="2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6" exitCode=0 Nov 03 08:28:54.223434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.223421 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" Nov 03 08:28:54.223625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.223469 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" event={"ID":"d8e89968-3a4b-4703-be77-07bfe931a99b","Type":"ContainerDied","Data":"2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6"} Nov 03 08:28:54.223625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.223504 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw" event={"ID":"d8e89968-3a4b-4703-be77-07bfe931a99b","Type":"ContainerDied","Data":"7ae61db8a742c17596e5889431b0deea9badde237a45251720c2d28cf4c092f8"} Nov 03 08:28:54.223625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.223520 5951 scope.go:122] "RemoveContainer" containerID="2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6" Nov 03 08:28:54.233218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.233159 5951 scope.go:122] "RemoveContainer" containerID="2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6" Nov 03 08:28:54.233518 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:54.233491 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6\": container with ID starting with 2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6 not found: ID does not exist" containerID="2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6" Nov 03 08:28:54.233611 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.233530 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6"} err="failed to get container status \"2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6\": rpc error: code = NotFound desc = could not find container \"2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6\": container with ID starting with 2bb0d617a2e41a7306cc1bb40d36c8da3919ca8da15484fbb2b87a74230ab2e6 not found: ID does not exist" Nov 03 08:28:54.248488 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.248465 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw"] Nov 03 08:28:54.251605 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.251582 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-w2-6bc553-kserve-84d89655b7-m7jnw"] Nov 03 08:28:54.619668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.618895 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw"] Nov 03 08:28:54.619668 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.619153 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" podUID="f7529824-f718-4ca9-97d8-a0f4529bff1a" containerName="maas-api" containerID="cri-o://009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8" gracePeriod=30 Nov 03 08:28:54.674133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.674101 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:28:54.741574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.741535 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" path="/var/lib/kubelet/pods/d8e89968-3a4b-4703-be77-07bfe931a99b/volumes" Nov 03 08:28:54.989551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:54.989527 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:28:55.180623 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.180591 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs\") pod \"f7529824-f718-4ca9-97d8-a0f4529bff1a\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " Nov 03 08:28:55.180848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.180731 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca\") pod \"f7529824-f718-4ca9-97d8-a0f4529bff1a\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " Nov 03 08:28:55.180848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.180765 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g\" (UniqueName: \"kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g\") pod \"f7529824-f718-4ca9-97d8-a0f4529bff1a\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " Nov 03 08:28:55.180848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.180805 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls\") pod \"f7529824-f718-4ca9-97d8-a0f4529bff1a\" (UID: \"f7529824-f718-4ca9-97d8-a0f4529bff1a\") " Nov 03 08:28:55.181205 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.181172 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca" pod "f7529824-f718-4ca9-97d8-a0f4529bff1a" (UID: "f7529824-f718-4ca9-97d8-a0f4529bff1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:28:55.183101 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.183072 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs" pod "f7529824-f718-4ca9-97d8-a0f4529bff1a" (UID: "f7529824-f718-4ca9-97d8-a0f4529bff1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:28:55.183671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.183617 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g" pod "f7529824-f718-4ca9-97d8-a0f4529bff1a" (UID: "f7529824-f718-4ca9-97d8-a0f4529bff1a"). InnerVolumeSpecName "kube-api-access-7xz8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:28:55.183769 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.183713 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls" pod "f7529824-f718-4ca9-97d8-a0f4529bff1a" (UID: "f7529824-f718-4ca9-97d8-a0f4529bff1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:28:55.232759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.232632 5951 generic.go:356] "Generic (PLEG): container finished" podID="f7529824-f718-4ca9-97d8-a0f4529bff1a" containerID="009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8" exitCode=0 Nov 03 08:28:55.232759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.232707 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" event={"ID":"f7529824-f718-4ca9-97d8-a0f4529bff1a","Type":"ContainerDied","Data":"009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8"} Nov 03 08:28:55.232759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.232718 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" Nov 03 08:28:55.232759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.232758 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw" event={"ID":"f7529824-f718-4ca9-97d8-a0f4529bff1a","Type":"ContainerDied","Data":"848ea544fc02fe77f71e671de91db7019cc953917a0b1b12b8ae49199ebd801f"} Nov 03 08:28:55.233108 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.232780 5951 scope.go:122] "RemoveContainer" containerID="009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8" Nov 03 08:28:55.243974 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.243953 5951 scope.go:122] "RemoveContainer" containerID="009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8" Nov 03 08:28:55.244261 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:28:55.244241 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8\": container with ID starting with 009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8 not found: ID does not exist" containerID="009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8" Nov 03 08:28:55.244318 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.244270 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8"} err="failed to get container status \"009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8\": rpc error: code = NotFound desc = could not find container \"009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8\": container with ID starting with 009153ffaf0a8f98435b7da17d29fa50f819c5cac80c97432b698132d407bed8 not found: ID does not exist" Nov 03 08:28:55.253980 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.253955 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw"] Nov 03 08:28:55.259607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.259587 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ffe7a8-5b56c6fd6f-vsbbw"] Nov 03 08:28:55.281932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.281912 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:55.281932 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.281932 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7529824-f718-4ca9-97d8-a0f4529bff1a-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:55.282048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.281942 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7529824-f718-4ca9-97d8-a0f4529bff1a-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:55.282048 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:55.281951 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xz8g\" (UniqueName: \"kubernetes.io/projected/f7529824-f718-4ca9-97d8-a0f4529bff1a-kube-api-access-7xz8g\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:28:56.737690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:28:56.737635 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7529824-f718-4ca9-97d8-a0f4529bff1a" path="/var/lib/kubelet/pods/f7529824-f718-4ca9-97d8-a0f4529bff1a/volumes" Nov 03 08:29:01.643720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:01.643686 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls"] Nov 03 08:29:01.644311 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:01.643942 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" containerID="cri-o://2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177" gracePeriod=30 Nov 03 08:29:01.958340 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:01.958301 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:29:02.613340 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:02.613301 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 08:29:03.054743 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.054713 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:29:03.068854 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.068812 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw\" (UniqueName: \"kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw\") pod \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " Nov 03 08:29:03.069030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.068866 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume\") pod \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\" (UID: \"46fecf4d-a101-4feb-9565-e7cb48fb80fe\") " Nov 03 08:29:03.069323 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.069294 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume" pod "46fecf4d-a101-4feb-9565-e7cb48fb80fe" (UID: "46fecf4d-a101-4feb-9565-e7cb48fb80fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:29:03.073785 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.073745 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw" pod "46fecf4d-a101-4feb-9565-e7cb48fb80fe" (UID: "46fecf4d-a101-4feb-9565-e7cb48fb80fe"). InnerVolumeSpecName "kube-api-access-gmtqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:03.170502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.170407 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmtqw\" (UniqueName: \"kubernetes.io/projected/46fecf4d-a101-4feb-9565-e7cb48fb80fe-kube-api-access-gmtqw\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:03.170502 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.170448 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fecf4d-a101-4feb-9565-e7cb48fb80fe-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:03.186675 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.186597 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c"] Nov 03 08:29:03.287073 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.287031 5951 generic.go:356] "Generic (PLEG): container finished" podID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerID="e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b" exitCode=0 Nov 03 08:29:03.287607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.287093 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" Nov 03 08:29:03.287607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.287114 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" event={"ID":"46fecf4d-a101-4feb-9565-e7cb48fb80fe","Type":"ContainerDied","Data":"e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b"} Nov 03 08:29:03.287607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.287189 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb" event={"ID":"46fecf4d-a101-4feb-9565-e7cb48fb80fe","Type":"ContainerDied","Data":"fb4b585dda97aff8ebfcd831a01be5b872127fb4489e21917dfd1253ce32791b"} Nov 03 08:29:03.287607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.287231 5951 scope.go:122] "RemoveContainer" containerID="e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b" Nov 03 08:29:03.287607 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.287533 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" containerID="cri-o://c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c" gracePeriod=30 Nov 03 08:29:03.307513 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.307491 5951 scope.go:122] "RemoveContainer" containerID="e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b" Nov 03 08:29:03.308731 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:03.307934 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b\": container with ID starting with e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b not found: ID does not exist" containerID="e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b" Nov 03 08:29:03.308731 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.307994 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b"} err="failed to get container status \"e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b\": rpc error: code = NotFound desc = could not find container \"e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b\": container with ID starting with e8f16f1c78b0b3b21d75cdb8e9aa6a612fe611e34e1013d522c7e75bab60085b not found: ID does not exist" Nov 03 08:29:03.324313 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.324285 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb"] Nov 03 08:29:03.332531 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:03.332502 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ffe7a8-b8688d8f9-74tvb"] Nov 03 08:29:04.736281 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:04.736249 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" path="/var/lib/kubelet/pods/46fecf4d-a101-4feb-9565-e7cb48fb80fe/volumes" Nov 03 08:29:05.690863 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:05.690832 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799454fcf4-24ntl"] Nov 03 08:29:05.691075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:05.691050 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-799454fcf4-24ntl" podUID="66b25251-3fe9-4716-b419-6d13474804f6" containerName="manager" containerID="cri-o://dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53" gracePeriod=30 Nov 03 08:29:05.945757 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:05.945699 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 08:29:06.000032 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:05.999996 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert\") pod \"66b25251-3fe9-4716-b419-6d13474804f6\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " Nov 03 08:29:06.000207 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.000097 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs\") pod \"66b25251-3fe9-4716-b419-6d13474804f6\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " Nov 03 08:29:06.000304 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.000215 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7\" (UniqueName: \"kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7\") pod \"66b25251-3fe9-4716-b419-6d13474804f6\" (UID: \"66b25251-3fe9-4716-b419-6d13474804f6\") " Nov 03 08:29:06.003059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.003028 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs" pod "66b25251-3fe9-4716-b419-6d13474804f6" (UID: "66b25251-3fe9-4716-b419-6d13474804f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:29:06.003182 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.003060 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7" pod "66b25251-3fe9-4716-b419-6d13474804f6" (UID: "66b25251-3fe9-4716-b419-6d13474804f6"). InnerVolumeSpecName "kube-api-access-xm4r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:06.006066 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.006042 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert" pod "66b25251-3fe9-4716-b419-6d13474804f6" (UID: "66b25251-3fe9-4716-b419-6d13474804f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:29:06.101609 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.101578 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4r7\" (UniqueName: \"kubernetes.io/projected/66b25251-3fe9-4716-b419-6d13474804f6-kube-api-access-xm4r7\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:06.101609 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.101609 5951 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-webhook-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:06.101609 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.101619 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b25251-3fe9-4716-b419-6d13474804f6-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:06.308165 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.308131 5951 generic.go:356] "Generic (PLEG): container finished" podID="66b25251-3fe9-4716-b419-6d13474804f6" containerID="dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53" exitCode=0 Nov 03 08:29:06.308370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.308175 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-24ntl" Nov 03 08:29:06.308370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.308224 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-24ntl" event={"ID":"66b25251-3fe9-4716-b419-6d13474804f6","Type":"ContainerDied","Data":"dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53"} Nov 03 08:29:06.308370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.308277 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-24ntl" event={"ID":"66b25251-3fe9-4716-b419-6d13474804f6","Type":"ContainerDied","Data":"ff624d8e9c2a44dddfe12597ff493f03ee00f566314b812f0c3234ea9abd759b"} Nov 03 08:29:06.308370 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.308306 5951 scope.go:122] "RemoveContainer" containerID="dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53" Nov 03 08:29:06.320206 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.320189 5951 scope.go:122] "RemoveContainer" containerID="dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53" Nov 03 08:29:06.320467 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:06.320449 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53\": container with ID starting with dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53 not found: ID does not exist" containerID="dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53" Nov 03 08:29:06.320524 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.320475 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53"} err="failed to get container status \"dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53\": rpc error: code = NotFound desc = could not find container \"dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53\": container with ID starting with dd2c388629b5e731aa87941d8e6886129b794c391e331d9855653566c28adf53 not found: ID does not exist" Nov 03 08:29:06.329224 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.329199 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799454fcf4-24ntl"] Nov 03 08:29:06.332103 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.332081 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-799454fcf4-24ntl"] Nov 03 08:29:06.742921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.742883 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b25251-3fe9-4716-b419-6d13474804f6" path="/var/lib/kubelet/pods/66b25251-3fe9-4716-b419-6d13474804f6/volumes" Nov 03 08:29:06.903363 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.903335 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:29:06.970223 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970140 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-799454fcf4-t4cpk"] Nov 03 08:29:06.970584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970554 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" Nov 03 08:29:06.970584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970564 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" Nov 03 08:29:06.970584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970576 5951 cpu_manager.go:414] "containerMap: removing container" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" Nov 03 08:29:06.970584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970581 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970593 5951 cpu_manager.go:414] "containerMap: removing container" podUID="f7529824-f718-4ca9-97d8-a0f4529bff1a" containerName="maas-api" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970598 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7529824-f718-4ca9-97d8-a0f4529bff1a" containerName="maas-api" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970604 5951 cpu_manager.go:414] "containerMap: removing container" podUID="693dc940-8e5b-4d53-94d5-06b60fca8bbe" containerName="payload-pre-processing" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970609 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693dc940-8e5b-4d53-94d5-06b60fca8bbe" containerName="payload-pre-processing" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970618 5951 cpu_manager.go:414] "containerMap: removing container" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970622 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970628 5951 cpu_manager.go:414] "containerMap: removing container" podUID="985bd2be-13e7-40a7-9022-38979241bc88" containerName="payload-processing" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970632 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985bd2be-13e7-40a7-9022-38979241bc88" containerName="payload-processing" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970660 5951 cpu_manager.go:414] "containerMap: removing container" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970667 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970680 5951 cpu_manager.go:414] "containerMap: removing container" podUID="1a01dbb7-a4c7-4393-9da1-72e7930db782" containerName="maas-api" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970687 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a01dbb7-a4c7-4393-9da1-72e7930db782" containerName="maas-api" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970698 5951 cpu_manager.go:414] "containerMap: removing container" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970702 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970710 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970714 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970720 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970724 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970729 5951 cpu_manager.go:414] "containerMap: removing container" podUID="66b25251-3fe9-4716-b419-6d13474804f6" containerName="manager" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970733 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b25251-3fe9-4716-b419-6d13474804f6" containerName="manager" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970742 5951 cpu_manager.go:414] "containerMap: removing container" podUID="388d8926-1579-4394-aa5f-fed4bee26395" containerName="maas-api" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970746 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388d8926-1579-4394-aa5f-fed4bee26395" containerName="maas-api" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970754 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" Nov 03 08:29:06.970752 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970758 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970764 5951 cpu_manager.go:414] "containerMap: removing container" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970769 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970775 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970779 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970785 5951 cpu_manager.go:414] "containerMap: removing container" podUID="533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970789 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970797 5951 cpu_manager.go:414] "containerMap: removing container" podUID="75476f61-7722-4160-b920-18d413090624" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970802 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75476f61-7722-4160-b920-18d413090624" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970808 5951 cpu_manager.go:414] "containerMap: removing container" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970812 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970818 5951 cpu_manager.go:414] "containerMap: removing container" podUID="68cc6679-e4d0-4740-bbac-233ed50166fa" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970821 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cc6679-e4d0-4740-bbac-233ed50166fa" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970828 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970832 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970839 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970843 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970851 5951 cpu_manager.go:414] "containerMap: removing container" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970855 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970860 5951 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d384d-4065-42f2-9334-42ca4377f1da" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970865 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d384d-4065-42f2-9334-42ca4377f1da" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970871 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a1598327-6f2f-47c8-963e-4587fbf725cf" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970875 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1598327-6f2f-47c8-963e-4587fbf725cf" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970883 5951 cpu_manager.go:414] "containerMap: removing container" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970887 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970893 5951 cpu_manager.go:414] "containerMap: removing container" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970897 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970903 5951 cpu_manager.go:414] "containerMap: removing container" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970907 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970913 5951 cpu_manager.go:414] "containerMap: removing container" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970916 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970922 5951 cpu_manager.go:414] "containerMap: removing container" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970927 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970933 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970938 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970944 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970949 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970955 5951 cpu_manager.go:414] "containerMap: removing container" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970959 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970966 5951 cpu_manager.go:414] "containerMap: removing container" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970970 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970978 5951 cpu_manager.go:414] "containerMap: removing container" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970982 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970988 5951 cpu_manager.go:414] "containerMap: removing container" podUID="f15ad4bc-6867-4027-b055-bf9b62894b9a" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.970992 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15ad4bc-6867-4027-b055-bf9b62894b9a" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971016 5951 cpu_manager.go:414] "containerMap: removing container" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971022 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971028 5951 cpu_manager.go:414] "containerMap: removing container" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971033 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971040 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971045 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971051 5951 cpu_manager.go:414] "containerMap: removing container" podUID="0ca589f5-8f4e-48b6-8642-44d214a72d28" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971055 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca589f5-8f4e-48b6-8642-44d214a72d28" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971061 5951 cpu_manager.go:414] "containerMap: removing container" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971065 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971070 5951 cpu_manager.go:414] "containerMap: removing container" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971075 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971082 5951 cpu_manager.go:414] "containerMap: removing container" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971087 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971093 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a64b095c-8435-44d7-8ca8-dab13c1f7aff" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971097 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64b095c-8435-44d7-8ca8-dab13c1f7aff" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971102 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971107 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971113 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971118 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971123 5951 cpu_manager.go:414] "containerMap: removing container" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971129 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971136 5951 cpu_manager.go:414] "containerMap: removing container" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971141 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971146 5951 cpu_manager.go:414] "containerMap: removing container" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971150 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971158 5951 cpu_manager.go:414] "containerMap: removing container" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971162 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971169 5951 cpu_manager.go:414] "containerMap: removing container" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971174 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971180 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971184 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971189 5951 cpu_manager.go:414] "containerMap: removing container" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971193 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971200 5951 cpu_manager.go:414] "containerMap: removing container" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971205 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971211 5951 cpu_manager.go:414] "containerMap: removing container" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971215 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971221 5951 cpu_manager.go:414] "containerMap: removing container" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971225 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971231 5951 cpu_manager.go:414] "containerMap: removing container" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971235 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971240 5951 cpu_manager.go:414] "containerMap: removing container" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971244 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971251 5951 cpu_manager.go:414] "containerMap: removing container" podUID="cdcdb196-e296-4f90-95c7-fb1136ac2885" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971255 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcdb196-e296-4f90-95c7-fb1136ac2885" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971262 5951 cpu_manager.go:414] "containerMap: removing container" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971266 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971274 5951 cpu_manager.go:414] "containerMap: removing container" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971278 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971284 5951 cpu_manager.go:414] "containerMap: removing container" podUID="710222a7-fae0-41ae-be1e-0312f29fd512" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971289 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710222a7-fae0-41ae-be1e-0312f29fd512" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971295 5951 cpu_manager.go:414] "containerMap: removing container" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971299 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971308 5951 cpu_manager.go:414] "containerMap: removing container" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971312 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971373 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87dd5f9-9463-42d1-89eb-91c0c287521a" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971384 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="74eefb2e-d52d-42f9-b10b-10f7efa5c038" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971389 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3eff667-e577-4878-bf46-4fc8a264aef0" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971394 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971401 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec249df-0e55-474e-ad96-e8a0769fd2bd" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971407 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca0f984-9823-43bd-a10e-6fd21931b505" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971413 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971418 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcaab243-c773-42cb-b917-d7ec04455bf1" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971422 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a561f60-a65e-4eeb-b7b2-b832818dc2bc" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971426 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971431 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7c1dda-db35-485e-8a09-a43d79f743d4" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971436 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcdb196-e296-4f90-95c7-fb1136ac2885" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971441 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971447 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ecfdea-2575-4a74-aeed-2a9068ae0d0f" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971452 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="710222a7-fae0-41ae-be1e-0312f29fd512" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971457 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="901fa1d8-7cb9-4d66-8a68-36970e80813a" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971464 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9415064-28a2-41a2-bfb8-0a06634ab720" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971469 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971474 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7529824-f718-4ca9-97d8-a0f4529bff1a" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971481 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="693dc940-8e5b-4d53-94d5-06b60fca8bbe" containerName="payload-pre-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971487 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5ba96c-79e3-4ed4-8fca-f42eb0d61ca0" containerName="main" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971492 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="985bd2be-13e7-40a7-9022-38979241bc88" containerName="payload-processing" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971498 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2df007-fdd1-4caa-b7e7-33477c96e02b" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971504 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a01dbb7-a4c7-4393-9da1-72e7930db782" containerName="maas-api" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971509 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971515 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed23459d-9aa8-492e-872d-5676cab351b4" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971519 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa862535-d1bb-41ad-88d0-aa1fe91b4fa3" containerName="istio-proxy" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971523 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b25251-3fe9-4716-b419-6d13474804f6" containerName="manager" Nov 03 08:29:06.971471 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971529 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="388d8926-1579-4394-aa5f-fed4bee26395" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971535 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5e3cdd-09d6-40d8-b479-1179b58a9c27" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971541 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2606b03-3902-4ea7-9eed-0dd6c35176e6" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971545 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb7caeb-48b0-4683-9dc2-08d96a124b64" containerName="payload-pre-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971550 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="533b1d8c-6f35-4f7c-8e09-13a26bd34ff6" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971556 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="75476f61-7722-4160-b920-18d413090624" containerName="payload-pre-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971562 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971567 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cc6679-e4d0-4740-bbac-233ed50166fa" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971572 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a351ffb-58a7-464c-a905-bcbfe06847ba" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971578 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fecf4d-a101-4feb-9565-e7cb48fb80fe" containerName="payload-pre-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971583 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d384d-4065-42f2-9334-42ca4377f1da" containerName="payload-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971588 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1598327-6f2f-47c8-963e-4587fbf725cf" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971593 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da169da-7303-4040-85c6-19d02852faa8" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971598 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a712fa-027b-4ece-81a9-a9a7c9c8c19e" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971602 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="1030fc0d-5e30-47a0-bd3e-9640d42e781c" containerName="payload-pre-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971606 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3588f6-d741-4e54-80dc-1ab1caa689ad" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971611 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd8f6e3-3644-469e-b73b-db7758756182" containerName="payload-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971616 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5472f604-f001-48f8-a95a-4680d038e507" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971621 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3b8e7c-fa60-4bbc-b919-5f7fcad02bf7" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971626 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8bdfd9-02f4-4661-b758-67cba3c395e0" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971631 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15ad4bc-6867-4027-b055-bf9b62894b9a" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971636 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="77624484-a1cd-47bc-a1f0-9c132a0da250" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971656 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e89968-3a4b-4703-be77-07bfe931a99b" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971661 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971666 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80ff06f-6f59-4664-9b6c-34a6272bf8eb" containerName="payload-pre-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971672 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca589f5-8f4e-48b6-8642-44d214a72d28" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971677 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="b310d0c8-f990-466f-b5bf-9a388124ef84" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971682 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1ca7f5-b4a8-4ce0-97e2-091d8b204cd4" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971687 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedc1ead-9210-438f-b524-aebf4b93942e" containerName="main" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971692 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64b095c-8435-44d7-8ca8-dab13c1f7aff" containerName="maas-api" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971697 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a65b48-df70-4abb-bf90-32d68094d841" containerName="payload-pre-processing" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971703 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7615b3e-68c7-4e1d-9230-30be0de2d060" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971707 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f909765-4f94-4242-bdda-9f9d8f345397" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971712 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd997040-85b1-4ebf-be01-04de25803871" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.971716 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="545ab5b3-977a-4507-909e-6fbaedc3877f" containerName="istio-proxy" Nov 03 08:29:06.976011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.974944 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:06.985212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.983316 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799454fcf4-t4cpk"] Nov 03 08:29:06.985212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.984469 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 08:29:06.985212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.984600 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k97gv\"" Nov 03 08:29:06.985212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.984729 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 08:29:06.985212 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:06.984756 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 08:29:07.007880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.007852 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.007898 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008037 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008074 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008126 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008189 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008159 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008316 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008389 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008636 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008460 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert\") pod \"86ebd2f2-96da-4da4-868b-e519288de041\" (UID: \"86ebd2f2-96da-4da4-868b-e519288de041\") " Nov 03 08:29:07.008888 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008683 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzjd\" (UniqueName: \"kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.008888 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008764 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.008994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008881 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:07.008994 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.008893 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.009110 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.009047 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.009432 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.009407 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:07.009560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.009533 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:29:07.009560 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.009548 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:07.009745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.009601 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:07.010594 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.010563 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "kube-api-access-7q4z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:07.011977 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.011949 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:29:07.012074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.012034 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:07.014982 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.014944 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy" pod "86ebd2f2-96da-4da4-868b-e519288de041" (UID: "86ebd2f2-96da-4da4-868b-e519288de041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:07.110492 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110458 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.110721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110562 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzjd\" (UniqueName: \"kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.110721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110612 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.110721 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110716 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110733 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110748 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110762 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ebd2f2-96da-4da4-868b-e519288de041-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110777 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ebd2f2-96da-4da4-868b-e519288de041-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110790 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q4z5\" (UniqueName: \"kubernetes.io/projected/86ebd2f2-96da-4da4-868b-e519288de041-kube-api-access-7q4z5\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110805 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.110893 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.110820 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ebd2f2-96da-4da4-868b-e519288de041-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:07.113632 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.113607 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.114745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.114719 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.118356 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.118334 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzjd\" (UniqueName: \"kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd\") pod \"maas-controller-799454fcf4-t4cpk\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.292292 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.292254 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:07.316267 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.316235 5951 generic.go:356] "Generic (PLEG): container finished" podID="86ebd2f2-96da-4da4-868b-e519288de041" containerID="2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177" exitCode=0 Nov 03 08:29:07.316397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.316357 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" Nov 03 08:29:07.316397 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.316375 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" event={"ID":"86ebd2f2-96da-4da4-868b-e519288de041","Type":"ContainerDied","Data":"2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177"} Nov 03 08:29:07.316514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.316446 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls" event={"ID":"86ebd2f2-96da-4da4-868b-e519288de041","Type":"ContainerDied","Data":"c2d5fec4d3451842e46386366fefc777ab146fda42390a73d88b0bb4700aff42"} Nov 03 08:29:07.316514 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.316475 5951 scope.go:122] "RemoveContainer" containerID="2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177" Nov 03 08:29:07.330404 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.330382 5951 scope.go:122] "RemoveContainer" containerID="8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33" Nov 03 08:29:07.342186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.342147 5951 scope.go:122] "RemoveContainer" containerID="2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177" Nov 03 08:29:07.344146 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:07.344098 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177\": container with ID starting with 2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177 not found: ID does not exist" containerID="2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177" Nov 03 08:29:07.344146 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.344142 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls"] Nov 03 08:29:07.344310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.344149 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177"} err="failed to get container status \"2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177\": rpc error: code = NotFound desc = could not find container \"2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177\": container with ID starting with 2d49b50f84d9c9c0805d525bdb0baef16416b500316937df4211fb6935919177 not found: ID does not exist" Nov 03 08:29:07.344310 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.344175 5951 scope.go:122] "RemoveContainer" containerID="8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33" Nov 03 08:29:07.344557 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:07.344528 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33\": container with ID starting with 8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33 not found: ID does not exist" containerID="8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33" Nov 03 08:29:07.344624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.344566 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33"} err="failed to get container status \"8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33\": rpc error: code = NotFound desc = could not find container \"8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33\": container with ID starting with 8dd293a25ea68648eb9cd98ee59dbfbfb60574bfdc6e4a473bf5a15c8cc49f33 not found: ID does not exist" Nov 03 08:29:07.346333 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.346306 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-jcbls"] Nov 03 08:29:07.582175 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.582144 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799454fcf4-t4cpk"] Nov 03 08:29:07.584900 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.584879 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:29:07.699921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.699874 5951 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:29:07.700092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:07.699934 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:29:08.327332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.327245 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-t4cpk" event={"ID":"ce57625a-a1d4-423b-b6ef-d63c8dd24efa","Type":"ContainerStarted","Data":"72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b"} Nov 03 08:29:08.327332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.327318 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-t4cpk" event={"ID":"ce57625a-a1d4-423b-b6ef-d63c8dd24efa","Type":"ContainerStarted","Data":"8c90944dff0245c31274b9a7569cb1bbb111adbc7592cd3e232521ea81fdd2c2"} Nov 03 08:29:08.327865 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.327497 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:08.344343 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.344289 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-799454fcf4-t4cpk" podStartSLOduration=1.8561683009999999 podStartE2EDuration="2.344271512s" totalImagesPullingTime="488.103211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:08.342601742 +0000 UTC m=+2087.724052670" watchObservedRunningTime="2026-11-03 08:29:08.344271512 +0000 UTC m=+2087.725722430" Nov 03 08:29:08.609059 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.609025 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729010 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729074 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729112 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729155 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729187 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729291 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729327 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729371 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.729674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.729431 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token\") pod \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\" (UID: \"497a28d8-1ebd-4c9e-ab47-5f1df765d997\") " Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.736775 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.737195 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.737441 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.740720 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.741573 5951 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istiod-ca-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.741594 5951 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.741611 5951 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-data\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.741626 5951 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-credential-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.750697 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.744953 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:08.778347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.771409 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:29:08.778347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.774740 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:08.778347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.775118 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:29:08.778347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.775592 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff" pod "497a28d8-1ebd-4c9e-ab47-5f1df765d997" (UID: "497a28d8-1ebd-4c9e-ab47-5f1df765d997"). InnerVolumeSpecName "kube-api-access-k6nff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:08.780307 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.780272 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ebd2f2-96da-4da4-868b-e519288de041" path="/var/lib/kubelet/pods/86ebd2f2-96da-4da4-868b-e519288de041/volumes" Nov 03 08:29:08.845768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.844635 5951 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-podinfo\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.845768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.844704 5951 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-envoy\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.845768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.844722 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6nff\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-kube-api-access-k6nff\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.845768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.844756 5951 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497a28d8-1ebd-4c9e-ab47-5f1df765d997-workload-socket\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:08.845768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:08.844771 5951 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497a28d8-1ebd-4c9e-ab47-5f1df765d997-istio-token\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:09.205398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.205367 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:29:09.263418 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.263382 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f\" (UniqueName: \"kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f\") pod \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " Nov 03 08:29:09.264145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.264118 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume\") pod \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\" (UID: \"addf2738-ce25-43f7-bae6-fc3d3a3c016c\") " Nov 03 08:29:09.265017 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.264994 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume" pod "addf2738-ce25-43f7-bae6-fc3d3a3c016c" (UID: "addf2738-ce25-43f7-bae6-fc3d3a3c016c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:29:09.267897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.267862 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f" pod "addf2738-ce25-43f7-bae6-fc3d3a3c016c" (UID: "addf2738-ce25-43f7-bae6-fc3d3a3c016c"). InnerVolumeSpecName "kube-api-access-gzv9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:09.354669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.352241 5951 generic.go:356] "Generic (PLEG): container finished" podID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerID="c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c" exitCode=0 Nov 03 08:29:09.354669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.352390 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" event={"ID":"497a28d8-1ebd-4c9e-ab47-5f1df765d997","Type":"ContainerDied","Data":"c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c"} Nov 03 08:29:09.354669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.352442 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" event={"ID":"497a28d8-1ebd-4c9e-ab47-5f1df765d997","Type":"ContainerDied","Data":"dfd1eb85c4879459258b7c2252a5b266a58f45d947ef1d0c1c869bc3c7a2a1aa"} Nov 03 08:29:09.354669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.352472 5951 scope.go:122] "RemoveContainer" containerID="c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c" Nov 03 08:29:09.354669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.352576 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c" Nov 03 08:29:09.360684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.356923 5951 generic.go:356] "Generic (PLEG): container finished" podID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerID="8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665" exitCode=0 Nov 03 08:29:09.360684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.357682 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" Nov 03 08:29:09.360684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.359035 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" event={"ID":"addf2738-ce25-43f7-bae6-fc3d3a3c016c","Type":"ContainerDied","Data":"8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665"} Nov 03 08:29:09.360684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.359082 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99" event={"ID":"addf2738-ce25-43f7-bae6-fc3d3a3c016c","Type":"ContainerDied","Data":"248fcf78a0a647fb036b758f7308333af1a441c37dfba15e61fbf48cd83d64eb"} Nov 03 08:29:09.365460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.365431 5951 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf2738-ce25-43f7-bae6-fc3d3a3c016c-plugins-config-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:09.365460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.365460 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzv9f\" (UniqueName: \"kubernetes.io/projected/addf2738-ce25-43f7-bae6-fc3d3a3c016c-kube-api-access-gzv9f\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:09.370573 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.370557 5951 scope.go:122] "RemoveContainer" containerID="d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6" Nov 03 08:29:09.378425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.378397 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c"] Nov 03 08:29:09.380250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.380230 5951 scope.go:122] "RemoveContainer" containerID="c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c" Nov 03 08:29:09.380585 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:09.380558 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c\": container with ID starting with c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c not found: ID does not exist" containerID="c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c" Nov 03 08:29:09.380816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.380592 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c"} err="failed to get container status \"c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c\": rpc error: code = NotFound desc = could not find container \"c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c\": container with ID starting with c370f8e0e07c532c8886fd743cf63db4104f7c9484c3b59f6eb8f238f7e8655c not found: ID does not exist" Nov 03 08:29:09.380816 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.380612 5951 scope.go:122] "RemoveContainer" containerID="d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6" Nov 03 08:29:09.381007 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:09.380939 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6\": container with ID starting with d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6 not found: ID does not exist" containerID="d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6" Nov 03 08:29:09.381055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.381019 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6"} err="failed to get container status \"d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6\": rpc error: code = NotFound desc = could not find container \"d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6\": container with ID starting with d29656abcc68b4471691d51716a506e5899aba5a26e42af467daf53f66e16ad6 not found: ID does not exist" Nov 03 08:29:09.381055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.381040 5951 scope.go:122] "RemoveContainer" containerID="8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665" Nov 03 08:29:09.383908 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.383887 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-bvm8c"] Nov 03 08:29:09.390494 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.390468 5951 scope.go:122] "RemoveContainer" containerID="8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665" Nov 03 08:29:09.390987 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:09.390959 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665\": container with ID starting with 8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665 not found: ID does not exist" containerID="8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665" Nov 03 08:29:09.391113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.390998 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665"} err="failed to get container status \"8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665\": rpc error: code = NotFound desc = could not find container \"8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665\": container with ID starting with 8149ba59e0037a4ceb6d141f5a0ea3a9f2be0a9f655e9abac2b6afe7cdf25665 not found: ID does not exist" Nov 03 08:29:09.392593 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.392571 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99"] Nov 03 08:29:09.418688 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:09.395913 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6bc553-59df79cd55-x8d99"] Nov 03 08:29:10.736011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:10.735978 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" path="/var/lib/kubelet/pods/497a28d8-1ebd-4c9e-ab47-5f1df765d997/volumes" Nov 03 08:29:10.736415 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:10.736374 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" path="/var/lib/kubelet/pods/addf2738-ce25-43f7-bae6-fc3d3a3c016c/volumes" Nov 03 08:29:19.360545 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:19.360508 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:21.840367 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:21.840340 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:29:21.840848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:21.840472 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:29:23.920238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920200 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb"] Nov 03 08:29:23.920768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920751 5951 cpu_manager.go:414] "containerMap: removing container" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" Nov 03 08:29:23.920823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920770 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" Nov 03 08:29:23.920823 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920818 5951 cpu_manager.go:414] "containerMap: removing container" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerName="payload-processing" Nov 03 08:29:23.920885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920824 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerName="payload-processing" Nov 03 08:29:23.920885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920846 5951 cpu_manager.go:414] "containerMap: removing container" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" Nov 03 08:29:23.920885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920850 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" Nov 03 08:29:23.920885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920867 5951 cpu_manager.go:414] "containerMap: removing container" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" Nov 03 08:29:23.920885 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920871 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d81aa6-a90e-4457-8dd9-41a95c107643" containerName="istio-proxy" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920946 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="addf2738-ce25-43f7-bae6-fc3d3a3c016c" containerName="payload-processing" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920956 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4853d6b-f6ba-4245-89aa-2069c46cea2a" containerName="istio-proxy" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920963 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ebd2f2-96da-4da4-868b-e519288de041" containerName="istio-proxy" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920969 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6aa698-50f9-4827-bfb2-3af5344613cd" containerName="istio-proxy" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920976 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920982 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b76e67a-2410-43d6-b272-2498ac21edc2" containerName="istio-proxy" Nov 03 08:29:23.921031 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.920990 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" Nov 03 08:29:23.924245 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.924220 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:23.940872 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.940809 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb"] Nov 03 08:29:23.943737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.943703 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4"] Nov 03 08:29:23.944335 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.944309 5951 cpu_manager.go:414] "containerMap: removing container" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" Nov 03 08:29:23.944421 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.944338 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a28d8-1ebd-4c9e-ab47-5f1df765d997" containerName="istio-proxy" Nov 03 08:29:23.947902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.947882 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:23.960033 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:23.960003 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4"] Nov 03 08:29:24.007580 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007548 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.007797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007589 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.007797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007605 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.007797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007737 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.007797 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007793 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007816 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkqt\" (UniqueName: \"kubernetes.io/projected/c3eac356-7b80-40d0-ab1c-db49382f2528-kube-api-access-slkqt\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007843 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007866 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eac356-7b80-40d0-ab1c-db49382f2528-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.007989 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfll\" (UniqueName: \"kubernetes.io/projected/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-kube-api-access-cpfll\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.008030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008025 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008082 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008118 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008176 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.008297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008202 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.008297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008246 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008297 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008269 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.008533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008341 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.008533 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.008374 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109347 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109311 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109374 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109406 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109431 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109466 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109499 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109554 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109530 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkqt\" (UniqueName: \"kubernetes.io/projected/c3eac356-7b80-40d0-ab1c-db49382f2528-kube-api-access-slkqt\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109568 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109603 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eac356-7b80-40d0-ab1c-db49382f2528-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109683 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfll\" (UniqueName: \"kubernetes.io/projected/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-kube-api-access-cpfll\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109709 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109735 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109758 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109775 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109791 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109814 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109839 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109867 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109920 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.109950 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.109922 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.110674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.110115 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.110674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.110207 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.110674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.110290 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.110674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.110360 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.110674 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.110573 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.110956 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.110759 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.111143 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.111122 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eac356-7b80-40d0-ab1c-db49382f2528-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.111334 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.111314 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.112737 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.112715 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.112849 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.112801 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.112959 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.112943 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.113023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.112975 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.116954 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.116923 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.117833 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.117802 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eac356-7b80-40d0-ab1c-db49382f2528-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.117920 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.117809 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkqt\" (UniqueName: \"kubernetes.io/projected/c3eac356-7b80-40d0-ab1c-db49382f2528-kube-api-access-slkqt\") pod \"maas-default-gateway-openshift-default-76f4876667-fvjb4\" (UID: \"c3eac356-7b80-40d0-ab1c-db49382f2528\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.117920 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.117909 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfll\" (UniqueName: \"kubernetes.io/projected/21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f-kube-api-access-cpfll\") pod \"maas-default-gateway-openshift-default-76f4876667-stldb\" (UID: \"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.242483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.242395 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:24.268934 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.268906 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:24.524613 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.524571 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb"] Nov 03 08:29:24.527922 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:29:24.527892 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ca9c2a_ccaa_49ce_ac6c_da232ae4b98f.slice/crio-f4cdc5ec5f5b3a1425fbb0e57cf33ef6891b8b46a0631631c5d10e364065dc2b WatchSource:0}: Error finding container f4cdc5ec5f5b3a1425fbb0e57cf33ef6891b8b46a0631631c5d10e364065dc2b: Status 404 returned error can't find the container with id f4cdc5ec5f5b3a1425fbb0e57cf33ef6891b8b46a0631631c5d10e364065dc2b Nov 03 08:29:24.530466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.529837 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:29:24.530466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.529913 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:29:24.530466 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.529952 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:29:24.557583 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.557556 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4"] Nov 03 08:29:24.561885 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:29:24.561854 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3eac356_7b80_40d0_ab1c_db49382f2528.slice/crio-1b22b967921e452674c8ed8adfa08e4b454cc7deb0778d268ddd61396044d9fd WatchSource:0}: Error finding container 1b22b967921e452674c8ed8adfa08e4b454cc7deb0778d268ddd61396044d9fd: Status 404 returned error can't find the container with id 1b22b967921e452674c8ed8adfa08e4b454cc7deb0778d268ddd61396044d9fd Nov 03 08:29:24.564174 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.564138 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:29:24.564237 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.564225 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:29:24.564285 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:24.564270 5951 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:29:25.455577 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:25.455538 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" event={"ID":"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f","Type":"ContainerStarted","Data":"ae380f638531ffdec3c81e9921dfce3d8705034df1a8e269c20520a7c0dca548"} Nov 03 08:29:25.456042 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:25.455602 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" event={"ID":"21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f","Type":"ContainerStarted","Data":"f4cdc5ec5f5b3a1425fbb0e57cf33ef6891b8b46a0631631c5d10e364065dc2b"} Nov 03 08:29:25.457144 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:25.457119 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" event={"ID":"c3eac356-7b80-40d0-ab1c-db49382f2528","Type":"ContainerStarted","Data":"e423896fd7591664389b70f059b958818dab280e785e5fe73fe21158753807c0"} Nov 03 08:29:25.457252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:25.457170 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" event={"ID":"c3eac356-7b80-40d0-ab1c-db49382f2528","Type":"ContainerStarted","Data":"1b22b967921e452674c8ed8adfa08e4b454cc7deb0778d268ddd61396044d9fd"} Nov 03 08:29:25.472390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:25.472310 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podStartSLOduration=2.472293779 podStartE2EDuration="2.472293779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:25.471846769 +0000 UTC m=+2104.853297686" watchObservedRunningTime="2026-11-03 08:29:25.472293779 +0000 UTC m=+2104.853744762" Nov 03 08:29:25.489733 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:25.489662 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podStartSLOduration=2.489632689 podStartE2EDuration="2.489632689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:25.489103727 +0000 UTC m=+2104.870554644" watchObservedRunningTime="2026-11-03 08:29:25.489632689 +0000 UTC m=+2104.871083605" Nov 03 08:29:26.244779 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:26.244733 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:26.245872 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:26.245840 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 08:29:26.246068 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:26.246041 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 08:29:26.269763 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:26.269740 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:26.271077 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:26.271051 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 08:29:26.271186 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:26.271094 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 08:29:27.244218 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:27.244177 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 08:29:27.244921 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:27.244867 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 08:29:27.269829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:27.269763 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 08:29:27.269829 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:27.269813 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 08:29:28.245720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:28.245318 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 08:29:28.245720 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:28.245373 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 08:29:28.269682 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:28.269627 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 08:29:28.269843 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:28.269708 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 08:29:29.252671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:29.252081 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 08:29:29.252671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:29.252146 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 08:29:29.270146 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:29.270103 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 08:29:29.270290 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:29.270168 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 08:29:30.249669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:30.249162 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 08:29:30.249669 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:30.249224 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 08:29:30.270241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:30.269934 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 08:29:30.270241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:30.269979 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 08:29:31.243589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:31.243141 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 08:29:31.243589 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:31.243200 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 08:29:31.269536 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:31.269498 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 08:29:31.269741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:31.269564 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 08:29:31.956508 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:31.956462 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:29:33.243880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:33.243841 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:29:33.244532 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:33.244497 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:29:33.269320 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:33.269263 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:29:33.269483 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:33.269329 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:29:34.287807 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:34.287764 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:29:34.288200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:34.287823 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:29:34.311836 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:34.311783 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:29:34.312145 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:34.312098 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:29:35.329897 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.329848 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-stldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:29:35.330439 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.329940 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" podUID="21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:29:35.354684 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.354615 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-fvjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:29:35.355142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.354984 5951 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" podUID="c3eac356-7b80-40d0-ab1c-db49382f2528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:29:35.525661 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.525613 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:35.526023 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.525996 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:35.526784 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.526763 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-stldb" Nov 03 08:29:35.544759 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.544727 5951 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:35.545926 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.545900 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:35.546442 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:35.546412 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-fvjb4" Nov 03 08:29:38.949830 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:38.949796 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799454fcf4-t4cpk"] Nov 03 08:29:38.950303 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:38.949988 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-799454fcf4-t4cpk" podUID="ce57625a-a1d4-423b-b6ef-d63c8dd24efa" containerName="manager" containerID="cri-o://72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b" gracePeriod=30 Nov 03 08:29:39.228619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.228587 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:39.260735 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.260697 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd\" (UniqueName: \"kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd\") pod \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " Nov 03 08:29:39.260928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.260759 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs\") pod \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " Nov 03 08:29:39.260928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.260840 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert\") pod \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\" (UID: \"ce57625a-a1d4-423b-b6ef-d63c8dd24efa\") " Nov 03 08:29:39.263241 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.263210 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs" pod "ce57625a-a1d4-423b-b6ef-d63c8dd24efa" (UID: "ce57625a-a1d4-423b-b6ef-d63c8dd24efa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:29:39.263881 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.263857 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd" pod "ce57625a-a1d4-423b-b6ef-d63c8dd24efa" (UID: "ce57625a-a1d4-423b-b6ef-d63c8dd24efa"). InnerVolumeSpecName "kube-api-access-tgzjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:29:39.264124 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.264101 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert" pod "ce57625a-a1d4-423b-b6ef-d63c8dd24efa" (UID: "ce57625a-a1d4-423b-b6ef-d63c8dd24efa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:29:39.362451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.362419 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:39.362451 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.362448 5951 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-webhook-cert\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:39.362680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.362459 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzjd\" (UniqueName: \"kubernetes.io/projected/ce57625a-a1d4-423b-b6ef-d63c8dd24efa-kube-api-access-tgzjd\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:29:39.550398 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.550359 5951 generic.go:356] "Generic (PLEG): container finished" podID="ce57625a-a1d4-423b-b6ef-d63c8dd24efa" containerID="72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b" exitCode=0 Nov 03 08:29:39.550584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.550411 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799454fcf4-t4cpk" Nov 03 08:29:39.550584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.550444 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-t4cpk" event={"ID":"ce57625a-a1d4-423b-b6ef-d63c8dd24efa","Type":"ContainerDied","Data":"72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b"} Nov 03 08:29:39.550584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.550522 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799454fcf4-t4cpk" event={"ID":"ce57625a-a1d4-423b-b6ef-d63c8dd24efa","Type":"ContainerDied","Data":"8c90944dff0245c31274b9a7569cb1bbb111adbc7592cd3e232521ea81fdd2c2"} Nov 03 08:29:39.550584 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.550543 5951 scope.go:122] "RemoveContainer" containerID="72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b" Nov 03 08:29:39.560390 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.560366 5951 scope.go:122] "RemoveContainer" containerID="72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b" Nov 03 08:29:39.560687 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:29:39.560666 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b\": container with ID starting with 72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b not found: ID does not exist" containerID="72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b" Nov 03 08:29:39.560768 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.560702 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b"} err="failed to get container status \"72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b\": rpc error: code = NotFound desc = could not find container \"72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b\": container with ID starting with 72dfa6fad48ab1ae270569a19c6c64bdf9fcf4e0125ff621afb04c960563db1b not found: ID does not exist" Nov 03 08:29:39.570064 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.570039 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799454fcf4-t4cpk"] Nov 03 08:29:39.572076 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:39.572052 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-799454fcf4-t4cpk"] Nov 03 08:29:40.736217 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:29:40.736184 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce57625a-a1d4-423b-b6ef-d63c8dd24efa" path="/var/lib/kubelet/pods/ce57625a-a1d4-423b-b6ef-d63c8dd24efa/volumes" Nov 03 08:30:01.955326 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:30:01.955290 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:30:18.569671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:30:18.565737 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:30:18.569671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:30:18.565858 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:30:31.954805 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:30:31.954727 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:31:01.955137 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:31:01.955097 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:31:32.099930 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:31:32.099884 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:32:01.956896 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:32:01.956849 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:32:31.954167 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:32:31.954127 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:33:01.955279 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:33:01.955235 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:33:31.954862 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:33:31.954774 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:34:01.953629 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:34:01.953583 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:34:21.876624 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:34:21.876594 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:34:21.879252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:34:21.879230 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:34:31.954480 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:34:31.954434 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:35:01.954874 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:35:01.954826 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:35:31.954957 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:35:31.954912 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:36:01.954292 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:36:01.954254 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:36:16.739680 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:16.739609 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth"] Nov 03 08:36:16.740264 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:16.739895 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" podUID="54ec4ebb-9864-4318-bbc4-39bef44cac21" containerName="manager" containerID="cri-o://35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392" gracePeriod=10 Nov 03 08:36:16.992522 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:16.992462 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 08:36:17.084544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.084435 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz\" (UniqueName: \"kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz\") pod \"54ec4ebb-9864-4318-bbc4-39bef44cac21\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " Nov 03 08:36:17.084544 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.084535 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume\") pod \"54ec4ebb-9864-4318-bbc4-39bef44cac21\" (UID: \"54ec4ebb-9864-4318-bbc4-39bef44cac21\") " Nov 03 08:36:17.085129 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.085104 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume" pod "54ec4ebb-9864-4318-bbc4-39bef44cac21" (UID: "54ec4ebb-9864-4318-bbc4-39bef44cac21"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:36:17.086928 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.086905 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz" pod "54ec4ebb-9864-4318-bbc4-39bef44cac21" (UID: "54ec4ebb-9864-4318-bbc4-39bef44cac21"). InnerVolumeSpecName "kube-api-access-89psz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:36:17.186053 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.186021 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89psz\" (UniqueName: \"kubernetes.io/projected/54ec4ebb-9864-4318-bbc4-39bef44cac21-kube-api-access-89psz\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:36:17.186053 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.186048 5951 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54ec4ebb-9864-4318-bbc4-39bef44cac21-extensions-socket-volume\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:36:17.534252 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.534211 5951 generic.go:356] "Generic (PLEG): container finished" podID="54ec4ebb-9864-4318-bbc4-39bef44cac21" containerID="35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392" exitCode=0 Nov 03 08:36:17.534434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.534255 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" event={"ID":"54ec4ebb-9864-4318-bbc4-39bef44cac21","Type":"ContainerDied","Data":"35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392"} Nov 03 08:36:17.534434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.534272 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" Nov 03 08:36:17.534434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.534308 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth" event={"ID":"54ec4ebb-9864-4318-bbc4-39bef44cac21","Type":"ContainerDied","Data":"d568cb9d777bc2a5b69bd75b219f77215f7053a014412d0087b9de27ff28d0f3"} Nov 03 08:36:17.534434 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.534331 5951 scope.go:122] "RemoveContainer" containerID="35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392" Nov 03 08:36:17.546248 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.546224 5951 scope.go:122] "RemoveContainer" containerID="35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392" Nov 03 08:36:17.546558 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:36:17.546538 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392\": container with ID starting with 35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392 not found: ID does not exist" containerID="35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392" Nov 03 08:36:17.546619 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.546571 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392"} err="failed to get container status \"35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392\": rpc error: code = NotFound desc = could not find container \"35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392\": container with ID starting with 35959f55d258d8a2873a6d9e2895738af69a39c327655d0542a44a836dcaf392 not found: ID does not exist" Nov 03 08:36:17.559142 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.559112 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth"] Nov 03 08:36:17.563379 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:17.563345 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q9cth"] Nov 03 08:36:18.735331 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:36:18.735301 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ec4ebb-9864-4318-bbc4-39bef44cac21" path="/var/lib/kubelet/pods/54ec4ebb-9864-4318-bbc4-39bef44cac21/volumes" Nov 03 08:36:31.955150 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:36:31.955058 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:37:01.954246 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:37:01.954195 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:37:17.686095 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686059 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k"] Nov 03 08:37:17.686677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686623 5951 cpu_manager.go:414] "containerMap: removing container" podUID="54ec4ebb-9864-4318-bbc4-39bef44cac21" containerName="manager" Nov 03 08:37:17.686677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686657 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ec4ebb-9864-4318-bbc4-39bef44cac21" containerName="manager" Nov 03 08:37:17.686677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686669 5951 cpu_manager.go:414] "containerMap: removing container" podUID="ce57625a-a1d4-423b-b6ef-d63c8dd24efa" containerName="manager" Nov 03 08:37:17.686677 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686674 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce57625a-a1d4-423b-b6ef-d63c8dd24efa" containerName="manager" Nov 03 08:37:17.686845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686763 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce57625a-a1d4-423b-b6ef-d63c8dd24efa" containerName="manager" Nov 03 08:37:17.686845 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.686780 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ec4ebb-9864-4318-bbc4-39bef44cac21" containerName="manager" Nov 03 08:37:17.689952 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.689932 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:17.694634 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.694613 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f6gvb\"" Nov 03 08:37:17.704057 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.704026 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k"] Nov 03 08:37:17.798902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.798862 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8s9\" (UniqueName: \"kubernetes.io/projected/c0f45fa5-6849-4e92-aec9-4b993cb446da-kube-api-access-vv8s9\") pod \"kuadrant-operator-controller-manager-678d87766f-8v45k\" (UID: \"c0f45fa5-6849-4e92-aec9-4b993cb446da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:17.799083 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.798931 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0f45fa5-6849-4e92-aec9-4b993cb446da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8v45k\" (UID: \"c0f45fa5-6849-4e92-aec9-4b993cb446da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:17.899876 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.899843 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8s9\" (UniqueName: \"kubernetes.io/projected/c0f45fa5-6849-4e92-aec9-4b993cb446da-kube-api-access-vv8s9\") pod \"kuadrant-operator-controller-manager-678d87766f-8v45k\" (UID: \"c0f45fa5-6849-4e92-aec9-4b993cb446da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:17.900047 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.899909 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0f45fa5-6849-4e92-aec9-4b993cb446da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8v45k\" (UID: \"c0f45fa5-6849-4e92-aec9-4b993cb446da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:17.900278 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.900258 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0f45fa5-6849-4e92-aec9-4b993cb446da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8v45k\" (UID: \"c0f45fa5-6849-4e92-aec9-4b993cb446da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:17.908018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:17.907998 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8s9\" (UniqueName: \"kubernetes.io/projected/c0f45fa5-6849-4e92-aec9-4b993cb446da-kube-api-access-vv8s9\") pod \"kuadrant-operator-controller-manager-678d87766f-8v45k\" (UID: \"c0f45fa5-6849-4e92-aec9-4b993cb446da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:18.003804 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:18.003729 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:18.289245 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:18.289214 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k"] Nov 03 08:37:18.290786 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:37:18.290762 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f45fa5_6849_4e92_aec9_4b993cb446da.slice/crio-fc004bdfaa0d76aa65247768db5340047762413341dd87afe3a62208207047ea WatchSource:0}: Error finding container fc004bdfaa0d76aa65247768db5340047762413341dd87afe3a62208207047ea: Status 404 returned error can't find the container with id fc004bdfaa0d76aa65247768db5340047762413341dd87afe3a62208207047ea Nov 03 08:37:18.859336 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:18.859291 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" event={"ID":"c0f45fa5-6849-4e92-aec9-4b993cb446da","Type":"ContainerStarted","Data":"95b8ff31790488af47a02ed7ec0dd28d1f0cb7db89aeefbebf9d957d17f1fa9c"} Nov 03 08:37:18.859739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:18.859356 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" event={"ID":"c0f45fa5-6849-4e92-aec9-4b993cb446da","Type":"ContainerStarted","Data":"fc004bdfaa0d76aa65247768db5340047762413341dd87afe3a62208207047ea"} Nov 03 08:37:18.859739 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:18.859398 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:18.878332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:18.878280 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" podStartSLOduration=1.8782665189999999 podStartE2EDuration="1.878266519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:37:18.876124385 +0000 UTC m=+2578.257575306" watchObservedRunningTime="2026-11-03 08:37:18.878266519 +0000 UTC m=+2578.259717430" Nov 03 08:37:29.867665 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:37:29.867612 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8v45k" Nov 03 08:37:31.955872 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:37:31.955827 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:38:01.954449 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:38:01.954410 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:38:31.955568 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:38:31.955527 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:39:01.955195 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:39:01.955153 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:39:21.929055 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:39:21.929022 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:39:21.933346 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:39:21.933325 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:39:31.963891 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:39:31.963801 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:40:01.953183 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:40:01.953134 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:40:31.954392 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:40:31.954351 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:41:01.955764 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:41:01.955722 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:41:02.069837 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.069796 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6"] Nov 03 08:41:02.073794 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.073770 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.086338 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.083442 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6"] Nov 03 08:41:02.231875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.231786 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f953394-9f83-48a7-8632-72c676d5a267-openshift-service-ca\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.231875 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.231827 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f953394-9f83-48a7-8632-72c676d5a267-metrics-certs\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.232079 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.231978 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f953394-9f83-48a7-8632-72c676d5a267-maas-api-tls\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.232079 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.232014 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95n2\" (UniqueName: \"kubernetes.io/projected/4f953394-9f83-48a7-8632-72c676d5a267-kube-api-access-v95n2\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.333250 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.333216 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f953394-9f83-48a7-8632-72c676d5a267-metrics-certs\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.333414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.333312 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f953394-9f83-48a7-8632-72c676d5a267-maas-api-tls\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.333414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.333350 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95n2\" (UniqueName: \"kubernetes.io/projected/4f953394-9f83-48a7-8632-72c676d5a267-kube-api-access-v95n2\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.333414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.333381 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f953394-9f83-48a7-8632-72c676d5a267-openshift-service-ca\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.334141 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.334083 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f953394-9f83-48a7-8632-72c676d5a267-openshift-service-ca\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.336000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.335969 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f953394-9f83-48a7-8632-72c676d5a267-metrics-certs\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.336000 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.335992 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f953394-9f83-48a7-8632-72c676d5a267-maas-api-tls\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.340219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.340194 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95n2\" (UniqueName: \"kubernetes.io/projected/4f953394-9f83-48a7-8632-72c676d5a267-kube-api-access-v95n2\") pod \"maas-api-6f6d4c9f4b-lpnx6\" (UID: \"4f953394-9f83-48a7-8632-72c676d5a267\") " pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.393851 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.393812 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:02.671818 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.671788 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6"] Nov 03 08:41:02.674332 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:41:02.674296 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f953394_9f83_48a7_8632_72c676d5a267.slice/crio-6ba290bb398492ab71559ed541c0709639f0284abe039bc820c98e7af713ba9f WatchSource:0}: Error finding container 6ba290bb398492ab71559ed541c0709639f0284abe039bc820c98e7af713ba9f: Status 404 returned error can't find the container with id 6ba290bb398492ab71559ed541c0709639f0284abe039bc820c98e7af713ba9f Nov 03 08:41:02.675747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.675726 5951 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:41:02.685253 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:02.685229 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" event={"ID":"4f953394-9f83-48a7-8632-72c676d5a267","Type":"ContainerStarted","Data":"6ba290bb398492ab71559ed541c0709639f0284abe039bc820c98e7af713ba9f"} Nov 03 08:41:03.692476 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:03.692436 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" event={"ID":"4f953394-9f83-48a7-8632-72c676d5a267","Type":"ContainerStarted","Data":"7a883d66b40a7bf5004486b57f0da50ac93a4cfd55d9ced9f5355d7d89f49ed3"} Nov 03 08:41:03.692905 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:03.692574 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:03.708728 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:03.708400 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" podStartSLOduration=1.2653443690000001 podStartE2EDuration="1.708386085s" totalImagesPullingTime="443.041716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:03.707551805 +0000 UTC m=+2803.089002721" watchObservedRunningTime="2026-11-03 08:41:03.708386085 +0000 UTC m=+2803.089837003" Nov 03 08:41:09.704274 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:09.704241 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f6d4c9f4b-lpnx6" Nov 03 08:41:09.750316 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:09.750275 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw"] Nov 03 08:41:09.750745 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:09.750683 5951 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" podUID="c028351b-059e-4929-98d3-0cce4713439b" containerName="maas-api" containerID="cri-o://f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55" gracePeriod=30 Nov 03 08:41:09.783238 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:09.783198 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" podUID="c028351b-059e-4929-98d3-0cce4713439b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.37:8443/health\": dial tcp 10.134.0.37:8443: connect: connection refused" Nov 03 08:41:11.022161 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.022137 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:41:11.033384 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.033359 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls\") pod \"c028351b-059e-4929-98d3-0cce4713439b\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " Nov 03 08:41:11.033493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.033408 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca\") pod \"c028351b-059e-4929-98d3-0cce4713439b\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " Nov 03 08:41:11.033493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.033443 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs\") pod \"c028351b-059e-4929-98d3-0cce4713439b\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " Nov 03 08:41:11.033493 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.033475 5951 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68\" (UniqueName: \"kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68\") pod \"c028351b-059e-4929-98d3-0cce4713439b\" (UID: \"c028351b-059e-4929-98d3-0cce4713439b\") " Nov 03 08:41:11.033868 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.033836 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca" pod "c028351b-059e-4929-98d3-0cce4713439b" (UID: "c028351b-059e-4929-98d3-0cce4713439b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:41:11.035613 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.035575 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls" pod "c028351b-059e-4929-98d3-0cce4713439b" (UID: "c028351b-059e-4929-98d3-0cce4713439b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:41:11.035719 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.035677 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs" pod "c028351b-059e-4929-98d3-0cce4713439b" (UID: "c028351b-059e-4929-98d3-0cce4713439b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:41:11.035909 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.035892 5951 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68" pod "c028351b-059e-4929-98d3-0cce4713439b" (UID: "c028351b-059e-4929-98d3-0cce4713439b"). InnerVolumeSpecName "kube-api-access-84q68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:41:11.134372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.134288 5951 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-maas-api-tls\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:41:11.134372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.134325 5951 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c028351b-059e-4929-98d3-0cce4713439b-openshift-service-ca\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:41:11.134372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.134336 5951 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c028351b-059e-4929-98d3-0cce4713439b-metrics-certs\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:41:11.134372 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.134344 5951 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84q68\" (UniqueName: \"kubernetes.io/projected/c028351b-059e-4929-98d3-0cce4713439b-kube-api-access-84q68\") on node \"ip-10-0-143-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 08:41:11.741366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.741332 5951 generic.go:356] "Generic (PLEG): container finished" podID="c028351b-059e-4929-98d3-0cce4713439b" containerID="f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55" exitCode=0 Nov 03 08:41:11.741366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.741371 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" event={"ID":"c028351b-059e-4929-98d3-0cce4713439b","Type":"ContainerDied","Data":"f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55"} Nov 03 08:41:11.741625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.741391 5951 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" Nov 03 08:41:11.741625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.741404 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw" event={"ID":"c028351b-059e-4929-98d3-0cce4713439b","Type":"ContainerDied","Data":"ab8ad3e928909d24fa249076c041db3633467dbb5712b235976ef866855d7ba4"} Nov 03 08:41:11.741625 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.741422 5951 scope.go:122] "RemoveContainer" containerID="f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55" Nov 03 08:41:11.752011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.751985 5951 scope.go:122] "RemoveContainer" containerID="f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55" Nov 03 08:41:11.752568 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:41:11.752548 5951 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55\": container with ID starting with f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55 not found: ID does not exist" containerID="f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55" Nov 03 08:41:11.752711 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.752576 5951 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55"} err="failed to get container status \"f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55\": rpc error: code = NotFound desc = could not find container \"f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55\": container with ID starting with f61f083dbe4a0ff4000d4fc74600bf3edd57ebacc6df57d489dfb7c5d19cfc55 not found: ID does not exist" Nov 03 08:41:11.762460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.762432 5951 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw"] Nov 03 08:41:11.769191 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:11.769163 5951 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7765546dfd-6vbkw"] Nov 03 08:41:12.735437 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:41:12.735407 5951 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c028351b-059e-4929-98d3-0cce4713439b" path="/var/lib/kubelet/pods/c028351b-059e-4929-98d3-0cce4713439b/volumes" Nov 03 08:41:31.956745 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:41:31.956700 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:42:01.955085 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:42:01.955047 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:42:31.954968 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:42:31.954882 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:43:01.954482 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:43:01.954444 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:43:31.955466 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:43:31.955423 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:43:48.564671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:43:48.563783 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-djh95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:43:48.564671 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:43:48.563918 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-djh95" podUID="9901831a-2efb-4fbc-a245-6c02ed4699de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:44:01.954524 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:44:01.954440 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:44:06.229332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:06.229070 5951 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sn42c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:44:06.229332 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:06.229129 5951 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sn42c" podUID="dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:44:21.969200 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:21.969169 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:44:21.972846 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:21.972826 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ww6bv_6c125505-d546-4704-9e55-ecb5b867aa4d/ovn-acl-logging/0.log" Nov 03 08:44:31.954164 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:44:31.954121 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:44:38.090187 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:38.090144 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c56f47d8c-lv8cv_13a7c571-1963-46f8-abd6-57b3cef729e6/manager/0.log" Nov 03 08:44:39.292366 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:39.292334 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzvxh_adcc9576-5d32-40f5-8c27-aa7a320ee7f1/manager/0.log" Nov 03 08:44:39.493422 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:39.493394 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rxd77_604ea4c3-1290-49a2-8a0a-135cce9cc58b/proxy/0.log" Nov 03 08:44:40.368873 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:40.368843 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6d4c9f4b-lpnx6_4f953394-9f83-48a7-8632-72c676d5a267/maas-api/0.log" Nov 03 08:44:43.909131 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:43.909091 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8v45k_c0f45fa5-6849-4e92-aec9-4b993cb446da/manager/0.log" Nov 03 08:44:44.316179 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:44.316148 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dzbtw_c9136285-1045-4ffc-9591-09176f766fd8/manager/0.log" Nov 03 08:44:45.236710 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:45.236667 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-nzds5_105f3437-0d08-43f6-a573-248cd4f36a96/istio-proxy/0.log" Nov 03 08:44:45.419176 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:45.419145 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wmqsr_b91f2daf-2470-4286-928b-c3fe71832aba/discovery/0.log" Nov 03 08:44:46.645071 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:46.645039 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-djh95_9901831a-2efb-4fbc-a245-6c02ed4699de/istio-proxy/0.log" Nov 03 08:44:46.822700 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:46.822671 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fvjb4_c3eac356-7b80-40d0-ab1c-db49382f2528/istio-proxy/0.log" Nov 03 08:44:47.355839 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:47.355806 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/1.log" Nov 03 08:44:47.513074 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:47.513035 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-stldb_21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f/istio-proxy/0.log" Nov 03 08:44:47.682741 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:47.682667 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk_7947a010-5e5c-462b-9739-6fd678b0c6ab/istio-proxy/0.log" Nov 03 08:44:47.846190 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:47.846158 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zbl64_63a234e8-a52f-40ca-89a3-d54337abfd1e/payload-pre-processing/0.log" Nov 03 08:44:48.012512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:48.012438 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:44:48.606962 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:48.606931 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/0.log" Nov 03 08:44:49.272783 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:49.272753 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m2gsg_797507d8-79e2-49b2-a51c-e72190a2fd67/main/0.log" Nov 03 08:44:49.611730 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:49.611701 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9rmr4_6cf247f2-4a20-4d4d-b55f-5fb8cd90b354/main/0.log" Nov 03 08:44:50.158438 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:50.158403 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99_9d85e397-5d01-4200-84d6-ccd89ffdb2a1/main/0.log" Nov 03 08:44:59.652343 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:44:59.652312 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c56f47d8c-lv8cv_13a7c571-1963-46f8-abd6-57b3cef729e6/manager/0.log" Nov 03 08:45:00.780015 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:00.779981 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzvxh_adcc9576-5d32-40f5-8c27-aa7a320ee7f1/manager/0.log" Nov 03 08:45:00.974512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:00.974478 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rxd77_604ea4c3-1290-49a2-8a0a-135cce9cc58b/proxy/0.log" Nov 03 08:45:01.483596 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:01.483556 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6d4c9f4b-lpnx6_4f953394-9f83-48a7-8632-72c676d5a267/maas-api/0.log" Nov 03 08:45:01.954840 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:45:01.954804 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:45:04.782269 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:04.782245 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8v45k_c0f45fa5-6849-4e92-aec9-4b993cb446da/manager/0.log" Nov 03 08:45:05.146319 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:05.146231 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dzbtw_c9136285-1045-4ffc-9591-09176f766fd8/manager/0.log" Nov 03 08:45:05.668906 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:05.668875 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-nzds5_105f3437-0d08-43f6-a573-248cd4f36a96/istio-proxy/0.log" Nov 03 08:45:05.838690 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:05.838633 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wmqsr_b91f2daf-2470-4286-928b-c3fe71832aba/discovery/0.log" Nov 03 08:45:07.049744 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:07.049707 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-djh95_9901831a-2efb-4fbc-a245-6c02ed4699de/istio-proxy/0.log" Nov 03 08:45:07.208957 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:07.208924 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fvjb4_c3eac356-7b80-40d0-ab1c-db49382f2528/istio-proxy/0.log" Nov 03 08:45:07.748334 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:07.748298 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/1.log" Nov 03 08:45:07.907197 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:07.907164 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-stldb_21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f/istio-proxy/0.log" Nov 03 08:45:08.079219 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:08.079184 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk_7947a010-5e5c-462b-9739-6fd678b0c6ab/istio-proxy/0.log" Nov 03 08:45:08.245425 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:08.245395 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zbl64_63a234e8-a52f-40ca-89a3-d54337abfd1e/payload-pre-processing/0.log" Nov 03 08:45:08.407460 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:08.407377 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:45:09.291574 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:09.291541 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m2gsg_797507d8-79e2-49b2-a51c-e72190a2fd67/main/0.log" Nov 03 08:45:09.610551 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:09.610456 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9rmr4_6cf247f2-4a20-4d4d-b55f-5fb8cd90b354/main/0.log" Nov 03 08:45:10.106583 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:10.106546 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kcr99_9d85e397-5d01-4200-84d6-ccd89ffdb2a1/main/0.log" Nov 03 08:45:31.954294 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:45:31.954249 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:45:44.148070 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:44.148035 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:45:44.315844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:45:44.315811 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:46:01.955127 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:46:01.955089 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:46:26.450965 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:26.450934 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:46:26.616060 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:26.616033 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:46:29.898811 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:29.898783 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hqxvr_abfaf52a-6e21-4290-93cf-367a0aa9d749/global-pull-secret-syncer/0.log" Nov 03 08:46:30.190540 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:30.190461 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-llmvz_728dc542-604c-46b2-9313-79f41aaf379e/konnectivity-agent/0.log" Nov 03 08:46:30.273621 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:30.273593 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-47.us-east-2.compute.internal_57afb19d8731b471ac7e459ac8d782fe/haproxy/0.log" Nov 03 08:46:31.954070 ip-10-0-143-47 kubenswrapper[5951]: E1103 08:46:31.954021 5951 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pp56g" containerName="extractor" Nov 03 08:46:37.521638 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:37.521601 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8v45k_c0f45fa5-6849-4e92-aec9-4b993cb446da/manager/0.log" Nov 03 08:46:37.621844 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:37.621812 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dzbtw_c9136285-1045-4ffc-9591-09176f766fd8/manager/0.log" Nov 03 08:46:39.026917 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.026863 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c6pqj_ce8c5d5b-8c05-4057-a77d-4a102186c208/kube-state-metrics/0.log" Nov 03 08:46:39.046011 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.045988 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c6pqj_ce8c5d5b-8c05-4057-a77d-4a102186c208/kube-rbac-proxy-main/0.log" Nov 03 08:46:39.070892 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.070864 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c6pqj_ce8c5d5b-8c05-4057-a77d-4a102186c208/kube-rbac-proxy-self/0.log" Nov 03 08:46:39.131809 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.131786 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4h9gh_2f8ebc2e-2b44-4837-b0ed-b9a4a897d52b/monitoring-plugin/0.log" Nov 03 08:46:39.315123 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.315051 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtg6x_84e304c5-3d08-4c75-8589-1ee0e67c4ed6/node-exporter/0.log" Nov 03 08:46:39.337140 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.337111 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtg6x_84e304c5-3d08-4c75-8589-1ee0e67c4ed6/kube-rbac-proxy/0.log" Nov 03 08:46:39.360168 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.360140 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtg6x_84e304c5-3d08-4c75-8589-1ee0e67c4ed6/init-textfile/0.log" Nov 03 08:46:39.436423 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.436384 5951 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj"] Nov 03 08:46:39.436963 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.436944 5951 cpu_manager.go:414] "containerMap: removing container" podUID="c028351b-059e-4929-98d3-0cce4713439b" containerName="maas-api" Nov 03 08:46:39.437084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.436964 5951 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c028351b-059e-4929-98d3-0cce4713439b" containerName="maas-api" Nov 03 08:46:39.437084 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.437071 5951 memory_manager.go:365] "RemoveStaleState removing state" podUID="c028351b-059e-4929-98d3-0cce4713439b" containerName="maas-api" Nov 03 08:46:39.440394 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.440373 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.448217 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.448196 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj"] Nov 03 08:46:39.490973 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.490938 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-sys\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.491133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.490980 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-proc\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.491133 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.491064 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-lib-modules\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.491220 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.491145 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kpg\" (UniqueName: \"kubernetes.io/projected/412c663d-3a37-459f-85f7-adfa09a6c81d-kube-api-access-h2kpg\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.491256 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.491241 5951 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-podres\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.498798 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.498777 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/prometheus/0.log" Nov 03 08:46:39.523614 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.523592 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/config-reloader/0.log" Nov 03 08:46:39.537115 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.537095 5951 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lsk8v\"/\"default-dockercfg-lfs26\"" Nov 03 08:46:39.540529 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.540511 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsk8v\"/\"openshift-service-ca.crt\"" Nov 03 08:46:39.540633 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.540592 5951 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsk8v\"/\"kube-root-ca.crt\"" Nov 03 08:46:39.551555 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.551536 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/thanos-sidecar/0.log" Nov 03 08:46:39.592030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.591976 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-podres\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592030 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592015 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-sys\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592040 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-proc\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592069 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-lib-modules\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592094 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-podres\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592101 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-sys\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592134 5951 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kpg\" (UniqueName: \"kubernetes.io/projected/412c663d-3a37-459f-85f7-adfa09a6c81d-kube-api-access-h2kpg\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592164 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592146 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-proc\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.592385 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.592168 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/412c663d-3a37-459f-85f7-adfa09a6c81d-lib-modules\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.598018 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.597999 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/kube-rbac-proxy-web/0.log" Nov 03 08:46:39.604747 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.604723 5951 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kpg\" (UniqueName: \"kubernetes.io/projected/412c663d-3a37-459f-85f7-adfa09a6c81d-kube-api-access-h2kpg\") pod \"perf-node-gather-daemonset-hktkj\" (UID: \"412c663d-3a37-459f-85f7-adfa09a6c81d\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.634855 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.634839 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/kube-rbac-proxy/0.log" Nov 03 08:46:39.660506 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.660485 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/kube-rbac-proxy-thanos/0.log" Nov 03 08:46:39.688246 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.688225 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7811fd3-c92a-4c78-86bd-c4c06334941c/init-config-reloader/0.log" Nov 03 08:46:39.755348 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.755322 5951 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:39.888373 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.888287 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f889895df-lctqt_aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/thanos-query/0.log" Nov 03 08:46:39.911204 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.911171 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f889895df-lctqt_aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/kube-rbac-proxy-web/0.log" Nov 03 08:46:39.937754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.937727 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f889895df-lctqt_aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/kube-rbac-proxy/0.log" Nov 03 08:46:39.960803 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.960764 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f889895df-lctqt_aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/prom-label-proxy/0.log" Nov 03 08:46:39.983512 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:39.983481 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f889895df-lctqt_aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/kube-rbac-proxy-rules/0.log" Nov 03 08:46:40.009927 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:40.009900 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f889895df-lctqt_aff1dba0-28ed-4e7f-ae34-9d30f7b881a1/kube-rbac-proxy-metrics/0.log" Nov 03 08:46:40.028462 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:40.028427 5951 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj"] Nov 03 08:46:40.031039 ip-10-0-143-47 kubenswrapper[5951]: W1103 08:46:40.031009 5951 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod412c663d_3a37_459f_85f7_adfa09a6c81d.slice/crio-99d9da2ffb9af9547ba25773af26366906545c2f1d5af0b4dfd949903e38d0f4 WatchSource:0}: Error finding container 99d9da2ffb9af9547ba25773af26366906545c2f1d5af0b4dfd949903e38d0f4: Status 404 returned error can't find the container with id 99d9da2ffb9af9547ba25773af26366906545c2f1d5af0b4dfd949903e38d0f4 Nov 03 08:46:40.746408 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:40.746366 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" event={"ID":"412c663d-3a37-459f-85f7-adfa09a6c81d","Type":"ContainerStarted","Data":"e36fe1d304f8541c143b1232e306f7935d212367aa039bfa6b4bed28a393eb91"} Nov 03 08:46:40.746603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:40.746422 5951 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" event={"ID":"412c663d-3a37-459f-85f7-adfa09a6c81d","Type":"ContainerStarted","Data":"99d9da2ffb9af9547ba25773af26366906545c2f1d5af0b4dfd949903e38d0f4"} Nov 03 08:46:40.746603 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:40.746510 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:40.761126 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:40.761070 5951 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" podStartSLOduration=1.761055475 podStartE2EDuration="1.761055475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:40.759569481 +0000 UTC m=+3140.141020398" watchObservedRunningTime="2026-11-03 08:46:40.761055475 +0000 UTC m=+3140.142506398" Nov 03 08:46:43.866864 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:43.866835 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zqtl5_5497a21d-0e9d-4191-bf7b-392785b052d0/dns/0.log" Nov 03 08:46:43.888795 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:43.888769 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zqtl5_5497a21d-0e9d-4191-bf7b-392785b052d0/kube-rbac-proxy/0.log" Nov 03 08:46:43.967414 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:43.967387 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbxj7_712391ec-a5a3-4824-9219-a1b56b5f214e/dns-node-resolver/0.log" Nov 03 08:46:44.584638 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:44.584611 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-skwds_30259dd3-c5f2-46b2-a5e1-acdfe2a77416/node-ca/0.log" Nov 03 08:46:45.616092 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:45.616059 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-nzds5_105f3437-0d08-43f6-a573-248cd4f36a96/istio-proxy/0.log" Nov 03 08:46:46.568501 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.568463 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wmqsr_b91f2daf-2470-4286-928b-c3fe71832aba/discovery/0.log" Nov 03 08:46:46.761820 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.761790 5951 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-hktkj" Nov 03 08:46:46.767339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.767314 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-djh95_9901831a-2efb-4fbc-a245-6c02ed4699de/istio-proxy/0.log" Nov 03 08:46:46.797448 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.797417 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fvjb4_c3eac356-7b80-40d0-ab1c-db49382f2528/istio-proxy/0.log" Nov 03 08:46:46.886516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.886426 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/0.log" Nov 03 08:46:46.887831 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.887812 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/1.log" Nov 03 08:46:46.912298 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.912275 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-stldb_21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f/istio-proxy/0.log" Nov 03 08:46:46.937880 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.937856 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk_7947a010-5e5c-462b-9739-6fd678b0c6ab/istio-proxy/0.log" Nov 03 08:46:46.962957 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.962938 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zbl64_63a234e8-a52f-40ca-89a3-d54337abfd1e/payload-pre-processing/0.log" Nov 03 08:46:46.990516 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:46.990495 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:46:49.137056 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:49.137024 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j4bwh_e8f0df62-afcf-4d4f-ae18-ddcdbdc99bba/serve-healthcheck-canary/0.log" Nov 03 08:46:49.733968 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:49.733883 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-nzds5_105f3437-0d08-43f6-a573-248cd4f36a96/istio-proxy/0.log" Nov 03 08:46:49.835578 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:49.835522 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pp56g_fba44899-0b4f-4ae4-86c0-a6e4657c9c73/kube-rbac-proxy/0.log" Nov 03 08:46:49.861606 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:49.861580 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pp56g_fba44899-0b4f-4ae4-86c0-a6e4657c9c73/exporter/0.log" Nov 03 08:46:49.886945 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:49.886913 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pp56g_fba44899-0b4f-4ae4-86c0-a6e4657c9c73/extractor/0.log" Nov 03 08:46:50.671311 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.671276 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wmqsr_b91f2daf-2470-4286-928b-c3fe71832aba/discovery/0.log" Nov 03 08:46:50.838467 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.838435 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-djh95_9901831a-2efb-4fbc-a245-6c02ed4699de/istio-proxy/0.log" Nov 03 08:46:50.876113 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.876081 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fvjb4_c3eac356-7b80-40d0-ab1c-db49382f2528/istio-proxy/0.log" Nov 03 08:46:50.946610 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.946525 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/0.log" Nov 03 08:46:50.947754 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.947734 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/1.log" Nov 03 08:46:50.962107 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.962081 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-stldb_21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f/istio-proxy/0.log" Nov 03 08:46:50.978339 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.978310 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk_7947a010-5e5c-462b-9739-6fd678b0c6ab/istio-proxy/0.log" Nov 03 08:46:50.993572 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:50.993546 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zbl64_63a234e8-a52f-40ca-89a3-d54337abfd1e/payload-pre-processing/0.log" Nov 03 08:46:51.009290 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:51.009268 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:46:52.635530 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:52.635498 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-22lvv_5f79fe08-451b-4072-b949-73db0bcf685c/manager/0.log" Nov 03 08:46:52.653500 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:52.653476 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-22lvv_5f79fe08-451b-4072-b949-73db0bcf685c/copy-manifests/0.log" Nov 03 08:46:53.899802 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:53.899770 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-5499fb4776-qx4xm_9777f6de-530f-40d3-88e9-ba27fc7b1b85/manager/0.log" Nov 03 08:46:53.927517 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:53.927489 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-zn6l4_6f99ae79-335d-4930-bb13-5cfe8914fa86/jobset-operator/0.log" Nov 03 08:46:54.120147 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:54.120117 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-nzds5_105f3437-0d08-43f6-a573-248cd4f36a96/istio-proxy/0.log" Nov 03 08:46:54.543575 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:54.543543 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c56f47d8c-lv8cv_13a7c571-1963-46f8-abd6-57b3cef729e6/manager/0.log" Nov 03 08:46:55.058075 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.058042 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wmqsr_b91f2daf-2470-4286-928b-c3fe71832aba/discovery/0.log" Nov 03 08:46:55.217209 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.217181 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-djh95_9901831a-2efb-4fbc-a245-6c02ed4699de/istio-proxy/0.log" Nov 03 08:46:55.236765 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.236737 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-fvjb4_c3eac356-7b80-40d0-ab1c-db49382f2528/istio-proxy/0.log" Nov 03 08:46:55.312173 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.312090 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/0.log" Nov 03 08:46:55.313848 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.313826 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sn42c_dfc1ad89-4520-43b0-b7c6-b7dcb10ee2b1/istio-proxy/1.log" Nov 03 08:46:55.328663 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.328620 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-stldb_21ca9c2a-ccaa-49ce-ac6c-da232ae4b98f/istio-proxy/0.log" Nov 03 08:46:55.347009 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.346988 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-b6bf76947-n7fnk_7947a010-5e5c-462b-9739-6fd678b0c6ab/istio-proxy/0.log" Nov 03 08:46:55.366902 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.366883 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zbl64_63a234e8-a52f-40ca-89a3-d54337abfd1e/payload-pre-processing/0.log" Nov 03 08:46:55.382821 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.382801 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-h9bfr_1705f8f5-bd72-4db8-8907-bcdb41ea65e4/payload-processing/0.log" Nov 03 08:46:55.957150 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.957116 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzvxh_adcc9576-5d32-40f5-8c27-aa7a320ee7f1/manager/0.log" Nov 03 08:46:55.983043 ip-10-0-143-47 kubenswrapper[5951]: I1103 08:46:55.983018 5951 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rxd77_604ea4c3-1290-49a2-8a0a-135cce9cc58b/proxy/0.log"